Cannot use importlib to import Cython extensions

[etgar...@gmail.com]

I have a module with functions that I compile without any manual optimizations. The normal version of the code looks like:

```py
def Main_Default(value_max: int, num_loops: int, rlock, runtime: str, compilation: str, call_type: str, subcall: str, case: str):
```

I then tried modifying the function to use C-type declarations like so:

```py
cdef void Main_Default(int value_max, int num_loops, rlock, str runtime, str compilation, str call_type, str subcall, str case):
```

The function doesn't return anything, and all the variables are of the correct type.

In my main program, I'm dynamically importing different modules using `importlib` based on a JSON file's parameters - basically letting a user choose what modules to import rather than importing all the modules. I do this through a snippet like this:

```py
if info["library"] in sys.modules: print("{0} module has already been loaded.".format(info["library"])) else: lib = importlib.import_module(info["library"], package=info["package"]) print("Loaded module '{0}'.".format(info["library"])) ... funcs[group] = vars(lib)["Main_Default"]
```

Where `info["package"]` is the top-level package of my module, and `info["library"]` is the actual module's full name. The last line is where I assign the `Main_Default` function to a dictionary value where I'll call it later.

All works fine in the first scenario where I'm using Python function declaration, where I'm met with the success message and the module runs fine when it's called.

The issue is that when using C-type function declarations, Importing the module works fine (as I still get the success message), but when trying to assign the `Main_Default` function of the module I get the following error message:

```py
Main_Default is not a valid function inside 'my_package.my_module'
```

I printed out the vars(lib) for my module just to make sure the function was listed, and the `Main_Default` function is not listed.
I believe this is due to `importlib` using the Python `import` function, while in this case I'd want to use Cython's `cimport` function.

I also tried changing the `cdef` to `cpdef` which should work, but then I'm met with this error:
```
cpdef void Main_Default(int value_max, int num_loops, rlock, str runtime, str compilation, str call_type, str subcall, str case): ^ ------------------------------------------------------------ my_package.my_module.pyx:14:6: closures inside cpdef functions not yet supported
``` 

Any way I can use `importlib` or some similar dynamic method for Cython extensions?

[Stefan Behnel]

etgar...@gmail.com schrieb am 18.07.20 um 21:39:

> I believe this is due to `importlib` using the Python `import` function,
> while in this case I'd want to use Cython's `cimport` function.

Not sure. You seem to be using the function from Python, so "import" and a
"def" or "cpdef" function seems appropriate.

Or are you using the function only from Cython code?

> I also tried changing the `cdef` to `cpdef` which should work, but then I'm
> met with this error:
> ```
> cpdef void Main_Default(int value_max, int num_loops, rlock, str runtime,
> str compilation, str call_type, str subcall, str case): ^
> ------------------------------------------------------------
> my_package.my_module.pyx:14:6: closures inside cpdef functions not yet
> supported
> ```

Use a "def" function instead. You can still use typed signatures for them.

Note that typing input arguments is not always a good idea since it
(surprise!) restricts the input types that users can pass into it, which
might get in the way of legitimate use cases. Only type input arguments if
there is a good reason.

In fact, you did not present any reason why you want a typed signature here.

Stefan

[etgar...@gmail.com]

In my program, I'm trying to compare regular Python code vs Python code that's compiled with Cython vs compiled Python code with manual static typing and other optimizations.

In this case, this is the manually optimized code where I'm trying to use C-type declarations wherever possible.

I suppose I can take the loss of cdef /cpdef function declarations and and use cdef/ cpdef for variables within the actual functions, but it would definitely make some impact on performance if I were able to make this a C function rather than a Python one.

The point I making was that importlib specifically uses "import" which only works on functions defined with "def" and doesn't work for functions defined with cdef or cpdef.

[Stefan Behnel]

etgar...@gmail.com schrieb am 18.07.20 um 22:51:

> The point I making was that importlib specifically uses "import" which only
> works on functions defined with "def" and doesn't work for functions
> defined with cdef or cpdef.

Your problem has nothing to do with "importlib". "import" imports modules,
not functions. The interface of a Python module (or extension module) is
Python objects.

There are a couple of ways to do something like this, depending on lots of
details about your specific use case that you didn't tell us about in your
descriptions. One thing you could try, for example, is to expose your
module interface function as an extension type that has a cdef method, and
call that method.

Stefan

[etgar...@gmail.com]

I'll actualyl provide the repo I'm working on for this project.
https://github.com/HunterAP23/Cython_Tester/tree/refactor
Should redirect to the "refactor" branch.

The files in question are src/Tester_Main.pyx that handles the module importing, and src/Primes/Optimized/Local/Standard.pyx is the file I'm basing this issue off of.

In the main part of the program, I loop through the Settings.json file and import the entirety of each module that should be imported.

I have a separate settings file called Settings_Optimized-Only.json which only imports that Standard.pyx file that I mentioned above.

The issue specifically happens on lines 187 to 190 of Tester_Main.pyx, where the "Main_Default" function does not appear in the list of items for that module when you import the src/Primes/Optimized/Local/Standard.pyx module.

[da-woods]

> In this case, this is the manually optimized code where I'm trying to
> use C-type declarations wherever possible.
> I suppose I can take the loss of cdef /cpdef function declarations and
> and use cdef/ cpdef for variables within the actual functions, but it
> would definitely make some impact on performance if I were able to
> make this a C function rather than a Python one.
>

You do know that def functions are still compiled by Cython and that the
speed of the code inside the def function is exactly the same as in a
cdef function?

The difference is that cdef functions have less overhead when called
from C or Cython. They're big advantage is for relatively short
functions that are called often.

Your "Main" functions do not look to be a case where cdef will make much
difference.