Extern void in pyx, with pxd, linking error "undefined reference to ..."

[Source]

Not sure if this is a bug or not, and how intended extern functions defined in Cython are.

But extern functions are highly useful when using C++ to call a Cython function.

I have just implemented a multithreaded asyncio.Protocol network protocol using this btw, with 10x performance increase (on par with Nginx as a web server) using Cython or 8x using a regular Python asyncio.Protocol class (only support for this type of lass so far, and no support for async def functions) with gil locked.

Only supports Linux (epoll), but yeah, point is extern functions are useful, but this limitation I just encountered is a little crippling, this makes it seemingly impossible to allow Cython cimport for that +2x bonus performance using Cython.

Here's a minimum replication of the issue:

a.pyx:

cdef extern void test():
        print("hello world")

a.pxd:

cdef extern void test()

b.pyx:

cimport a
a.test()

Best regards, Source.

[Source]

Sorry, I just discovered this doesn't happen with cythonize, only when I use my custom "cycompile" script which uses cython and g++ with $(python3-config) directly (I loathe distutils setup, I use cython way too much to write a distutils loader/config thingie for every cython code I want to compile).

Since cythonize is no longer verbose or supports being verbose though, I'm not sure what I can do to fix this, but I can try experiment and see if I can figure something out.

Would be nice if cythonize supported an embed option to compile an executable instead of a library so I wouldn't have to write my own compile script, or that cythonize supported more verbose compilation like it used to do before.

I guess what I'll have to do for now is install an older version of cython to get cythonize to be more verbose.

Either way, thanks for the job you guys do, I still find that Cython is the most powerful language out there for now, or at the very least is my favorite one.

[Source]

After recognizing the aforementioned issue with extern functions and an issue with cimporting functions as probable bugs I decided to create an issue for those here https://github.com/cython/cython/issues/6514

The project/module I'm referring to in the issue that I mentioned there has now been fully implemented (test phase/pre-alpha only) that demonstrates the usefulness of these extern functions defined in Cython.

Here's the benchmarks of my network code (with the use of Python including a lot of overhead from this where the data is sent from Python) compared to other modules/references on an i9-9900k with 8 CPU threads, my module only being possible to make with multiple extern void functions to call Cython from C++ without using Python's C API:

Keep in mind the performance scales with the number of CPU threads. I tried the same code on my Intel Ultra 7 laptop with 22 threads, and it more than doubled the performance compared to these 8 threads!

Connection: keep-alive (does not work for all types of servers):

Asyncio: 52671 requests/sec (static HTTP data string)

Uvloop: 93790 requests/sec (static HTTP data string)

Nginx (8 threads): 179909 requests/sec (index.html)

My module (8 threads): 287275 requests/sec (static HTTP data string)

Connection: close:

Asyncio: 9523 requests/s

Uvloop: 24532 requests/s

My module: 85641 requests/s

[Source]

Sorry for posting again, but after I posted the first issue I realized there's a second issue, which I have a question about that I asked here https://github.com/cython/cython/issues/6533#issuecomment-2522490294 that might be more of a support or broader question about Cython design.

Namely: So is the behavior laid out in the design of Cython that it will by default allow import and cimport modules in sys.path default locations, but not in the current location?

What's the rationale behind this design decision, when Python does not work this way and allows importing any module in CWD and subdirectories by default (given there's an ``__init__.py`` file in the subdirectory)?

Additionally how are Cython developers as a result supposed to test cimport functionality before releasing a module with cimport as a public feature in mind?

[Source]

Just want to announce/update the bench results while waiting for a response that the module I made reached 1.2M HTTP requests on my 16 CPU core laptop @ 100% CPU clock using keep-alive + desync mode (which I came up with later, improving performance by 2-3x + my laptop has a faster CPU, explaining the difference in results from earlier) using a static HTTP page.

Though even with an actual open source HTTP server (Cypache) I'm writing which parses HTTP headers, gets the URL request, resolves the pathname, checks if the URL contains the www dir in full, etc, it should still actually achieve exactly 1M requests +/- since on 50% underclocked CPU performance I get up to exactly 500k requests per second.

Mentioning this mainly for 2 reasons:

1. To make the point that extern functions to call from C++ to Cython are absolutely invaluable imo.

2. I know that Stefan wrote a blog post about a similar (but in my opinion fairly weaker) result by Nexedi in 2018 on his blog http://blog.behnel.de/posts/fast-python-web-server-with-cython.html and I thought my results were worth mentioning in that regard for exceeding these results by a fair amount and making the HTTP server itself open source (fastepoll, the backend model, is currently proprietary with a copyright to myself though, but can be substituted for with uvloop very easily).

Although Cypache does real processing, reading from index.html or any HTML file, supporting index traversal, calling realpath to resolve paths, etc, it uses heavy caching (with an update timer of up to 1 sec) for basically everything to achieve these high-end results.