AttributeError: 'builtin_function_or_method' object has no attribute '__globals__'

2,838 views
Skip to first unread message

walt3k

unread,
Mar 16, 2022, 4:26:19 PM3/16/22
to cython-users
Hi there,

I am a novice cython/python user so this question may have an obvious answer but I've not been able to find it!   We are using both python and cython-based "Tests" for verifying hardware using an open source tool called cocotb.  Each of our python programs are considered "a testcase" which we need to store the file name for.  We support both python testcases and cython testcases where the user puts our "test" decorator in front of the function in the python file/cython file which is the testcase.   In the python case, we can get the full filename for our testcase we are running but we get an error with cython.  Here is the decorator up to the red line that produces the  AttributeError  which is copied below:

def test():
    def call_coro(coro):
        currentTestName=coro.__name__
        @functools.wraps(coro)
        async def wrapper(*args, **kwargs):
            filename="UNKNOWN_BECAUSE_THIS_IS_CYTHON"
            # cython/cocofu_ext tests don't have this. Need to find another way.
            filename=coro.__globals__['__file__']


ERROR    Test Failed: test_force_pass_stops_busy_rtx (result was AttributeError)
                      Traceback (most recent call last):
                        File "__init__.py", line 83, in wrapper
                          filename=coro.__globals__['__file__']
                      AttributeError: 'builtin_function_or_method' object has no attribute '__globals__'

I certainly don't have to get the full filename to the decorated test file using __globals__ if there is another way.  I would like to understand though why cython is different and if differences like this one are documented or can be inferred.  

Thanks for the help,  Walt.

walt3k

unread,
Mar 16, 2022, 4:34:56 PM3/16/22
to cython-users
BTW, if I "cimport" the module which defines this decorator, I then get the error

@cocofu.test()
^
------------------------------------------------------------

/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/test_calc1.pyx:117:0: cimported module has no attribute 'test'

Stefan Behnel

unread,
Mar 16, 2022, 4:48:03 PM3/16/22
to cython...@googlegroups.com
Hi,

walt3k schrieb am 16.03.22 um 21:22:
> but we get an error with cython. Here is the decorator up to the red line
> that produces the AttributeError which is copied below:
>
> def test():
> def call_coro(coro):
> currentTestName=coro.__name__
> @functools.wraps(coro)
> async def wrapper(*args, **kwargs):
> filename="UNKNOWN_BECAUSE_THIS_IS_CYTHON"
> # cython/cocofu_ext tests don't have this. Need to find another
> way.
> *filename=coro.__globals__['__file__']*
>
>
> ERROR Test Failed: test_force_pass_stops_busy_rtx (result was
> AttributeError)
> Traceback (most recent call last):
> File "__init__.py", line 83, in wrapper
> filename=coro.__globals__['__file__']
> AttributeError: 'builtin_function_or_method' object
> has no attribute '__globals__'

You have to enable the "binding" directive. That's still disabled by
default in Cython 0.29.x. Cython 3.0 has it on automatically, so you can
also use that instead.

Stefan

Stefan Behnel

unread,
Mar 16, 2022, 4:51:07 PM3/16/22
to cython...@googlegroups.com
walt3k schrieb am 16.03.22 um 21:32:
> BTW, if I "cimport" the module which defines this decorator, I then get the
> error
>
> @cocofu.test()
> ^
> ------------------------------------------------------------
>
> /u/walt3k/work/cocotb/new_controls/Calc1Cocotb/test_calc1.pyx:117:0:
> cimported module has no attribute 'test'

That's because "test" isn't something that the .pxd file exports. It's part
of the Python interface of the module (I guess).

If you want to use both interfaces, you have to import and cimport both.

Stefan

walt3k

unread,
Mar 16, 2022, 6:03:17 PM3/16/22
to cython-users
I tried both importing and cimporting the module with defines that decorator in __init__.py and still had the same error.

Is that what you intended me to try or did I misunderstand?

Thx,  Walt.

walt3k

unread,
Mar 16, 2022, 6:03:23 PM3/16/22
to cython-users
I installed version 3.0a1 (BTW - Would that be the best choice if I want the best stable version?)  and it works thx!!! -- but sort of.

The problem is I am asking for  __globals__['__file__']   which for python tests is the .py name of the file. 

Ideally I'd like to get the .pyx filename for cython tests but instead I get the name of the .so file which cython created when it compiled the .pyx file.

Is there a way in my decorator to get the name of the .pyx file which was the source file used to create the .so file reported by   __globals__['__file__']  ?

Thx again,  Walt.

Stefan Behnel

unread,
Mar 16, 2022, 6:08:35 PM3/16/22
to cython...@googlegroups.com
walt3k schrieb am 16.03.22 um 22:39:
> On Wednesday, March 16, 2022 at 3:51:07 PM UTC-5 Stefan Behnel wrote:
>> walt3k schrieb am 16.03.22 um 21:32:
>>> BTW, if I "cimport" the module which defines this decorator, I then get
>> the
>>> error
>>>
>>> @cocofu.test()
>>> ^
>>> ------------------------------------------------------------
>>>
>>> /u/walt3k/work/cocotb/new_controls/Calc1Cocotb/test_calc1.pyx:117:0:
>>> cimported module has no attribute 'test'
>>
>> That's because "test" isn't something that the .pxd file exports. It's
>> part
>> of the Python interface of the module (I guess).
>>
>> If you want to use both interfaces, you have to import and cimport both.
>
> I tried both importing and cimporting the module with defines that
> decorator in __init__.py and still had the same error.
>
> Is that what you intended me to try or did I misunderstand?

cocotb does not seem to be using Cython, so there is nothing to cimport
from it. Just use import.

Stefan

Stefan Behnel

unread,
Mar 16, 2022, 6:17:36 PM3/16/22
to cython...@googlegroups.com
walt3k schrieb am 16.03.22 um 23:02:
> On Wednesday, March 16, 2022 at 3:48:03 PM UTC-5 Stefan Behnel wrote:
>> walt3k schrieb am 16.03.22 um 21:22:
>>> but we get an error with cython. Here is the decorator up to the red line
>>> that produces the AttributeError which is copied below:
>>>
>>> def test():
>>> def call_coro(coro):
>>> currentTestName=coro.__name__
>>> @functools.wraps(coro)
>>> async def wrapper(*args, **kwargs):
>>> filename="UNKNOWN_BECAUSE_THIS_IS_CYTHON"
>>> # cython/cocofu_ext tests don't have this. Need to find another
>>> way.
>>> *filename=coro.__globals__['__file__']*
>>>
>>>
>>> ERROR Test Failed: test_force_pass_stops_busy_rtx (result was
>>> AttributeError)
>>> Traceback (most recent call last):
>>> File "__init__.py", line 83, in wrapper
>>> filename=coro.__globals__['__file__']
>>> AttributeError: 'builtin_function_or_method' object
>>> has no attribute '__globals__'
>>
>> You have to enable the "binding" directive. That's still disabled by
>> default in Cython 0.29.x. Cython 3.0 has it on automatically, so you can
>> also use that instead.
>
> I installed version 3.0a1 (BTW - Would that be the best choice if I want
> the best stable version?) and it works thx!!! -- but sort of.
>
> The problem is I am asking for __globals__['__file__'] which for python
> tests is the .py name of the file.
>
> Ideally I'd like to get the .pyx filename for cython tests but instead I
> get the name of the .so file which cython created when it compiled the .pyx
> file.

That's intentional. It gives you the file that implements the module. And
for a binary module, that's the .so file.


> Is there a way in my decorator to get the name of the .pyx file which was
> the source file used to create the .so file reported by
> __globals__['__file__'] ?

Is that really the file you need? It's not needed at runtime, just for the
translation. However, you'd probably still want to ship that file with your
binary package (read, wheel) in order to get proper traceback code lines
printed for exceptions.

If you keep the .py or .pyx file next to the .so file, then it should be
easy to find it from "__file__", by just removing everything behind the
first dot, and then appending the right extension. Even if you don't know
the right extension, you can probably just guess and look it up in the file
system (and cache the final path in a dict if that's really too slow).

Stefan

walt3k

unread,
Mar 16, 2022, 6:26:09 PM3/16/22
to cython-users
You are right, cocotb does not use cython.  My library code which "plugs in" cocotb into my company's environment does because we need to interact with C++ code.  So I have a python library which has a decorator which the people in my company have to use to "plug in."  They can use this decorator either in a pure python file or in a cython pyx file where they are connecting to C++ code/etc.  So it's not a cocotb issue at all I think - it's my issue with my library - called "cocofu" if that makes the communication easier.  I didn't come up with that name BTW...  :-)  

Thanks for helping,  Walt.

walt3k

unread,
Mar 16, 2022, 6:40:00 PM3/16/22
to cython-users
Thanks for the feedback.  Indeed, it is the file I need because it defines the "testcase" which passed or failed.  If the .so file gets moved to another location and became disassociated with the source pyx file, I'd like to know what input file was used to create the .so.   Once compiled, it does not appear the pyx file is required to execute.
 I thought there would be a way to get the full path name to the source pyx file because if I run the strings command on my cython-compiled .so file on a linuxX86/gcc platform, the full filename is in there as a symbol.  I thought for sure there would be a way to get it via an api.  I can just dig it out directly but of course know that's a risky proposition since cython could change.

thx again,  Walt.

walt3k

unread,
Mar 16, 2022, 6:46:46 PM3/16/22
to cython-users
I just realized I did not read your response carefully.  I see I need to keep the pyx file with the .so file for the traceback support you mentioned.  That is good information.

Ignore my previous reply.

Thx again Walt.

walt3k

unread,
Mar 17, 2022, 3:15:01 AM3/17/22
to cython-users
Sorry to come back again but I'd like to know what traceback info is lost if the pyx file is no longer in place. 

I have an assert firing in a cython test which appears to print correctly when I move the .pyx file to a different directory not associated with the test/compile/etc.  For example, this AssertionError is being thrown and printed correctly even with the pyx file gone

Cyc 0000000: ERROR    Test Failed: test_wait_for_execute_loop_via_event (result was AssertionError)

                      Traceback (most recent call last):
                        File "/afs/apd.pok.ibm.com/u/walt3k/work/cocotb/new_controls/cocofu/src/cocofu/__init__.py", line 101, in wrapper
                          return await coro(*args, **kwargs)
                        File "test_calc1.pyx", line 56, in test_wait_for_execute_loop_via_event
                      AssertionError

Moreover, I am also catching this exception in a try/except block in my decorator which then raises the exception. Before doing so, the code collects the following information which also appears to be the same as when the pyx file is in place:

        # we need to know what line in the test() caused this e to happen
        # so look back in the traceback to get the correct file/line/etc.
        TestFrameSummary=traceback.extract_tb(e.__traceback__)[1]
        inData['full_filename']=TestFrameSummary.filename
        inData['filename']=os.path.basename(inData['full_filename'])
        inData['lineno']=TestFrameSummary.lineno
        inData['funcname']=TestFrameSummary.name
        inData['etype']=type(e)
        inData['e']=e

So what in the traceback should not work correctly now that the pyx file is gone?

thx for all the help,  Walt.

Stefan Behnel

unread,
Mar 17, 2022, 3:25:02 AM3/17/22
to cython...@googlegroups.com
walt3k schrieb am 17.03.22 um 01:38:
> Sorry to come back again but I'd like to know what traceback info is lost
> if the pyx file is no longer in place.
>
> I have an assert firing in a cython test which appears to print correctly
> when I move the .pyx file to a different directory not associated with the
> test/compile/etc. For example, this AssertionError is being thrown and
> printed correctly even with the pyx file gone
>
> Cyc 0000000: ERROR Test Failed: test_wait_for_execute_loop_via_event
> (result was AssertionError)
> Traceback (most recent call last):
> File
> "/afs/apd.pok.ibm.com/u/walt3k/work/cocotb/new_controls/cocofu/src/cocofu/__init__.py",
> line 101, in wrapper
> return await coro(*args, **kwargs)
> File "test_calc1.pyx", line 56, in
> test_wait_for_execute_loop_via_event
> AssertionError

Well, if you remove the source file, then there is no access to the source
code any more.

The source code in the last stack frame is not printed, because it's simply
not available if you remove the .pyx file.

Similarly, line coverage reporting etc. will be hindered by not being able
to show the source code.

Stefan

walt3k

unread,
Mar 22, 2022, 1:49:52 AM3/22/22
to cython-users
Hi Stefan (and all),

Although I have not removed the .pyx file, I'm experiencing the problem you described Stefan which is the traceback is not printing any info coming from my .pyx file.  It stops at the last .py file which had the decorator used in the .pyx file.  I have copied the .pyx file to different location like $PWD where I am running, to the same directory where the .so file exists - no luck.   Like I mentioned, if I run the strings command on the .so file I can see the literal path name to the .pyx file in the .so file as a string.  I also see the short name for the .pyx file.

Do you have any ideas or advice on how to get the traceback to show the .pyx lines?  Thx a bunch.  Walt.

walt3k

unread,
Mar 22, 2022, 7:07:46 PM3/22/22
to cython-users
More information I think I probably should have shared in my last post - apologies.  I am assuming how I compile/link and where the compilation takes place and what options are passed could matter.  

I am not compiling the cython-generated .C file into an executable.  It is getting compiled into a .o file and then a .so file which is loaded at runtime.  This is because we are using cocotb as I mentioned previously.  The cython .pyx file is essentially "a testcase" we are running in a larger environment which is simulating physical computer chips of various sorts.  So here is the cmake command executed to compile a file called "test_calc1.pyx".
(The variable name ${rtx_name} would be "test_calc1")

add_custom_command(OUTPUT ${rtx_name}_wrap.C
    COMMAND $ENV{COCOFUHOME}/python_bin/cython -v -3 --cplus -X c_string_type=str -X c_string_encoding=utf8
        -I $ENV{COCOFUHOME}/src/cocofu_ext -I $ENV{COCOFUHOME}/src -I $ENV{COCOFUHOME}/src/cocofu_ext/parm
        ${src_dir}/${rtx_name}.pyx -o ${rtx_name}_wrap.C
                DEPENDS ${src_dir}/${rtx_name}.pyx ${ARGN}
                COMMENT Generating Cython wrapper for ${rtx_name}
                VERBATIM
                COMMAND_EXPAND_LISTS)


In the generated test_calc1_wrap.C I see

static const char *__pyx_f[] = {
  "test_calc1.pyx",
};


and

static const char __pyx_k_afs_apd_pok_ibm_com_u_walt3k_wo[] = "/afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/test_calc1.pyx";

Here's the verbose compile output:

for the cython call:
cd /afs/apdn/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles/src && /afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/python_bin/cython -v -3 --cplus -X c_string_type=str -X c_string_encoding=utf8 -I /afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/src/cocofu_ext -I /afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/src -I /afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/src/cocofu_ext/parm /afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/test_calc1.pyx -o test_calc1_wrap.C

compiling the .o

cd /afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles/src && /opt/xsite/cte/tools/simarama/gcc/bin/g++ 
-D_AIB2_MAX_SIZE_=256 -D_AIB2_NODEID_8BITS_ -D_Z_PBC_DIRINT_SUPPORT_ -I/opt/xsite/cte/tools/fusion/prod/plugins/cml/R85_0/312.0/src/cml
-I/afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/src -I/opt/xsite/cte/tools/simarama/latest 
-I/afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles/src
 -isystem /opt/xsite/cte/tools/fusion/prod/R85_0/src -I/afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/src -std=c++11 
-fmessage-length=0 -fPIC -m64 -D__64BIT__ -O1 -DNDEBUG=0 -Wall -Wno-unused-macros -Wno-unused-function -Wno-unused-local-typedefs -Wno-sign-compare -Wno-maybe-uninitialized -Wno-unused-but-set-variable -I/afs/apd/func/vlsi/eclipz/common/verif/cocotb_python/include/python3.9 
-I/afs/apd/func/vlsi/eclipz/common/verif/cocotb_python/include/python3.9 -Wno-unused-result -Wsign-compare -Wall -MD 
-MT src/CMakeFiles/test_calc1_o.dir/test_calc1_wrap.C.o -MF CMakeFiles/test_calc1_o.dir/test_calc1_wrap.C.o.d 
-o CMakeFiles/test_calc1_o.dir/test_calc1_wrap.C.o
-c /afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles/src/test_calc1_wrap.C


linking the .so 
make[4]: Entering directory `/afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles'
[100%] Linking CXX shared module ../../test_calc1.so
cd /afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles/src && /afs/eda/tools/apps/cmake/3.22.1/x86_64/bin/cmake -E cmake_link_script CMakeFiles/test_calc1.dir/link.txt --verbose=1
/opt/xsite/cte/tools/simarama/gcc/bin/g++ -fPIC 
-Wl,-L/afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/build/amd64_linux26_RH7,-lcocofu_ptr,--enable-new-dtags,
-rpath=/afs/apd/u/walt3k/work/cocotb/new_controls/cocofu/build/amd64_linux26_RH7 -fPIC -Wl,-Bsymbolic -Wl,--allow-shlib-undefined -m64  
-shared  -o ../../test_calc1.so CMakeFiles/test_calc1_o.dir/test_calc1_wrap.C.o  -L/afs/apd/func/vlsi/eclipz/common/verif/cocotb_python/lib 
-lcrypt -lpthread -ldl -lutil -lm
make[4]: Leaving directory `/afs/apd/u/walt3k/work/cocotb/new_controls/Calc1Cocotb/rtx/build/amd64_linux26_RH7/CMakeFiles'
[100%] Built target test_calc1


I have scattered the .pyx file all over those directories and no matter where I put it, the tracback still does not include lines from the .pyx file.

Any thoughts/advice would be much appreciated.   Thx,  Walt.







Reply all
Reply to author
Forward
0 new messages