DLL not found for C++ Python Extension
35 views
Skip to first unread message
Hana Ali
Jun 12, 2023, 1:51:27 PM6/12/23
to cython-users
Hello,
I've been stuck at this problem for a while now. I'm currently creating an extension module for Python called simulations. This extension is written in C++. For this extension, I needed to use the gsl library, and link it in my setup.py file. For some reason, if I don't link to the library in libraries, library_dirs, the code doesn't work because it's missing libraries, but if I do, I get the error ImportError: DLL load failed while importing simulations: The specified module could not be found. I'm not sure if they're related at all, but if I run the program without gsl, it works fine.
This is my setup.py:
# Building an extension module from C source
from distutils.core import setup, Extension
import numpy
simulation_mod = Extension('simulations',
sources = ['simulation_main.cpp'],
language="c++",
include_dirs=[numpy.get_include(),
"C:\\Users\\shahi\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\numpy\\core\\include",
"C:\\cpp_libs\\include",
"C:\\cpp_libs\\include\\bayesopt\\include",
"C:\\src\\vcpkg\\installed\\x64-windows\\include",
"C:\\src\\vcpkg\\installed\\x86-windows\\include",
],
library_dirs=[
"C:\\src\\vcpkg\\installed\\x64-windows\\lib",
"C:\\src\\vcpkg\\installed\\x86-windows\\lib"
],
libraries=[
"gsl",
"gslcblas"
]
)
# The main setup command
setup(name = 'DissertationModels',
version="1.0",
description="Simulation models and temporal integration for different model",
ext_modules=[simulation_mod],
# py_modules=['simulation_interface'],
)
my tasks.json
{
"tasks": [
{
"type": "cppbuild",
"label": "C/C++: cpp.exe build active file",
"command": "C:\\msys64\\ucrt64\\bin\\cpp.exe",
"args": [
"-fdiagnostics-color=always",
"-g",
"${file}",
"-o",
"${fileDirname}\\${fileBasenameNoExtension}.exe",
"-IC:\\src\\vcpkg\\installed\\x64-windows\\include",
"-IC:\\src\\vcpkg\\installed\\x86-windows\\include",
"-IC:\\src\\vcpkg\\installed\\x86-windows\\include\\python3.10",
"-IC:\\Users\\shahi\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\numpy\\core\\include",
"-IC:\\cpp_libs\\include",
"-IC:\\cpp_libs\\include\\bayesopt\\include"
],
"options": {
"cwd": "${fileDirname}"
},
"problemMatcher": [
"$gcc"
],
"group": {
"kind": "build",
"isDefault": true
},
"detail": "Task generated by Debugger."
},
{
"type": "cppbuild",
"label": "C/C++: g++.exe build active file",
"command": "C:\\msys64\\ucrt64\\bin\\g++.exe",
"args": [
"-fdiagnostics-color=always",
"-g",
"${file}",
"-o",
"${fileDirname}\\${fileBasenameNoExtension}.exe",
"${fileDirname}\\${fileBasenameNoExtension}.exe",
"-IC:\\src\\vcpkg\\installed\\x64-windows\\include",
"-IC:\\src\\vcpkg\\installed\\x86-windows\\include",
"-IC:\\src\\vcpkg\\installed\\x86-windows\\include\\python3.10",
"-IC:\\Users\\shahi\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\numpy\\core\\include",
"-IC:\\cpp_libs\\include",
"-IC:\\cpp_libs\\include\\bayesopt\\include"
],
"options": {
"cwd": "${fileDirname}"
},
"problemMatcher": [
"$gcc"
],
"group": "build",
"detail": "Task generated by Debugger."
}
],
"version": "2.0.0"
}
Both setup.py build and install don't give any errors - they run as usual. Not sure why suddenly getting this error, but it's frustrating! Any help is appreciated! I've never built an extension before :)
Thanks,
Hana
mattip
Jun 13, 2023, 3:18:36 PM6/13/23
to cython-users
Welcome to the wonderful world of mixing C, system libraries, and Python. It can be frustrating sometimes just figuring out where to even look for an answer.
There are two parts to successfully completing your task:
- finding headers and link libs when building the c-extension module
- finding the DLL at runtime when importing the c-extension module.
It seems you have successfully completed part 1. To make sure the module can load, you need to make sure the needed DLL is on the dll search path [0] at runtime. For python3.8+, you can use `os.add_dll_directory(<path>)` before the `import simulations` statement. For earlier python versions, and some versions of anaconda python, you need to add it to the PATH before starting the interpreter. Another option that should JustWork(tm) is to copy the DLL so it can be found along side the c-extension module in the same directory. A third option is to add a manifest to your project, which is probably the most secure but the most complicated option.
Matti
On Monday, 12 June 2023 at 20:51:27 UTC+3 hana...@gmail.com wrote:
Hello,I've been stuck at this problem for a while now. I'm currently creating an extension module for Python called simulations. This extension is written in C++. For this extension, I needed to use the gsl library, and link it in my setup.py file. For some reason, if I don't link to the library in libraries, library_dirs, the code doesn't work because it's missing libraries, but if I do, I get the error ImportError: DLL load failed while importing simulations: The specified module could not be found. I'm not sure if they're related at all, but if I run the program without gsl, it works fine.
...
Hana Ali
Jun 13, 2023, 4:16:58 PM6/13/23
to cython-users
Hey mattip,
Thank you for your response! I've only been working with this for ~2 weeks, but I've already learnt so much about compilers/linkers/etc. that it almost (almost) makes the frustration worth it.
My problem has actually changed now. I essentially used lucasg's Dependencies to find which DLLs are missing from the created .pyd file, and added all the missing ones to my PATH. The thing is, even though I have no missing DLLs from the dependent modules (at least not according to the .exe), I now get the error: ImportError: DLL load failed while importing simulations: The specified module could not be found regardless of what I do.
I run py setup.py build > py setup.py install, and when I run py -m pip list -v, I can only see DissertationModules installed, and no simulations (the former is the name I have in setup() as well as the PyModuleDef below, the latter is the first argument in Extension). I don't understand why the install process doesn't actually work, even though it outputs that installation is performed. I've checked that I'm using the same python path for everything, but I can't figure out what the problem is.
I really appreciate your response! This is a project for my MSc dissertation, and I'm not sure who to ask :/
Thank you,
Hana
Hana Ali
Jun 13, 2023, 4:17:15 PM6/13/23
to cython-users
Hey mattip,
I'm not sure if my last message got posted, so I'm rewriting this one.
Thank you so much for your response! I've only been working with this for ~2 weeks, but I've already learnt so much about compilers/linkers/etc., and it's really fun :) My problem's actually gotten worse somehow - instead of working without gsl, now the program doesn't work at all. I keep getting the error ImportError: DLL load failed while importing simulations: The specified module could not be found. Following the post above, I checked for the dependent DLLs using lucasg's Dependencies, and noticed I had 2 missing (python311.dll and gsl.dll), which I then added to my PATH.
Now, when I run setup.py build > setup.py install, and then check py -m pip list -v, I notice that simulations (the name of the module) is not in the list, but DissertationModels is. Is this expected behaviour? The path links to where simulations.py and simulations.cp311-win_amd64 are, but I'm not sure if simulations is supposed to appear with pip. I think maybe this has something to do with why I'm getting the error? I didn't post it earlier, but if it helps, here is my PyModuleDef:
// Function that wraps these functions into methods of a module
static PyMethodDef IntegrationMethods[] = {
// { // BOLD model
// "electrical_to_bold",
// electrical_to_bold,
// METH_VARARGS,
// "Solves the BOLD model equations, and returns BOLD activity"
// },
{
"wilson_model",
parsing_wilson_inputs,
METH_VARARGS,
"Solves the Wilson-Cowan model equations, and returns electrical activity"
},
{ // Sentinel to properly exit function
NULL, NULL, 0, NULL
}
};
// Function that wraps the methods in a module
static struct PyModuleDef SimulationsModule = {
PyModuleDef_HEAD_INIT,
"simulations",
"Module containing functions for simulation compiled in C",
-1,
IntegrationMethods
};
// Function that creates the modules
PyMODINIT_FUNC PyInit_simulations(void)
{
import_array();
Py_Initialize();
return PyModule_Create(&SimulationsModule);
}
I checked that I am using the same python for both the pip install and for running the program (also removed the python3.10 includePath in tasks and cpp_properties), so I'm not really sure what the problem can be. It's a bit frustrating because I don't really know who to ask for help irl, but I'm hoping you could help me fix it!
Thank you so much for taking the time :) Have a great day!
Thanks,
Hana
On Tuesday, June 13, 2023 at 8:18:36 PM UTC+1 mattip wrote:
mattip
Jun 14, 2023, 6:28:15 AM6/14/23
to cython-users
Dependencies is great, but it does not exactly mimic the mechanism used to find additional DLLs if used outside the python process. Even when used inside the process, sometimes it can give faulty results, due to the complicated ways windows can find and load DLLs. I have in the past put the a python command to run it (`subprocess.run(["Dependencies", ...], ...) in the code just before the line with "import simulations". Then you should see that "python311.dll" is already loaded into the process, since you are running Python.
The problem is more gsl.dll. Adding its location to the PATH, as I said before, will only work on some versions of python. On other versions, python can ignore the PATH when searching for DLLs. You then have to use `os.add_dll_directory()` instead. See the documentation https://docs.python.org/3/library/os.html#os.add_dll_directory and have "fun" diving into the other links there. I think the directory where the PYD file is located is always searched, so if you just want to get going, and gsl.dll has no further dependencies, you might have some success copying gsl.dll to the directory with the PYD file.
It would be nice if Windows provided a debug option for the dll search process, which would verbosely print out where it is looking, instead of the "DLL load failed" message.
Matti
On Tuesday, 13 June 2023 at 23:17:15 UTC+3 hana...@gmail.com wrote:
Hey mattip,I'm not sure if my last message got posted, so I'm rewriting this one.Thank you so much for your response! I've only been working with this for ~2 weeks, but I've already learnt so much about compilers/linkers/etc., and it's really fun :) My problem's actually gotten worse somehow - instead of working without gsl, now the program doesn't work at all. I keep getting the error ImportError: DLL load failed while importing simulations: The specified module could not be found. Following the post above, I checked for the dependent DLLs using lucasg's Dependencies, and noticed I had 2 missing (python311.dll and gsl.dll), which I then added to my PATH.
Hana Ali
Jun 14, 2023, 4:14:08 PM6/14/23
to cython-users
Hey mattip,
Thank you so much for all your help :) Unfortunately adding them to where simulations installs (in the DissertationModels folder) didn't fix the problem :/ Could there be any other reason why it isn't working?
Have a great day, and take care!
Thank you,
Hana
Hana
mattip
Jun 15, 2023, 6:48:32 AM6/15/23
to cython-users
I fear we may be drifting off the purpose of this group. Maybe you could ping me (@mattip) on an issue in your public github repo. But one last thing: did you make sure all the dependent DLLs are copied? I see there is a gslcblas dependency: does that in turn depend on finding openblas.dll or so? Or maybe you are using the cygwin build of gsl and it depends on finding some cygwin runtime libraries? In this case, running "Dependencies" on your dll is a great tool to help you figure out what else you need to copy.
Matti
On Wednesday, 14 June 2023 at 23:14:08 UTC+3 hana...@gmail.com wrote:
Hey mattip,
Reply all
Reply to author
Forward
0 new messages