Importing Python extension module (.pyd) at runtime, not packaged in .exe

[Mark Melvin]

Hi All,

I have an application that requires a folder of files that is effectively an external "SDK" that can change independent of my app, and there is a Python binding for that SDK in the form of a compiled C extension (sd.pyd).

I have written my script to dynamically resolve and import this module at runtime and it all works great from Python. I can point it to a folder somewhere via a command line argument and it will augment sys.path and PATH accordingly, and successfully import the module.

When I turn my script into and executable with pyInstaller, it fails to import my module with:

ImportError: DLL load failed while importing sd: The specified module could not be found

I have tried manually setting the PATH and PYTHONPATH in the console and even copying the .pyd file to be alongside the .exe itself, but nothing works. Now the weird part. If I re-run the pyInstaller build with my PYTHONPATH set to the location where my .pyd file resides, it creates an executable that *does* work. So it appears that pyInstaller has its own notion of a PYTHONPATH that can be augmented at build time but not runtime??

I do not want to bake this path into the executable, nor do I want to build the binary file into the .exe itself because it depends on the external SDK version I am referencing and I would like this to be dynamic. But how to I tell a pyInstaller'd .exe about it at runtime?? Here is the code that is doing the dynamic import in my Python module:

_WIN_SAMPLES_PATH = 'samples/win/bin'

def __get_run_path():

    """Returns the directory of this script"""

    if getattr(sys, 'frozen', False):

        # we are running in a bundle

        #bundle_dir = sys._MEIPASS

        return os.path.abspath(os.path.dirname(sys.executable))

    # we are running in a normal Python environment

    #bundle_dir = os.path.dirname(os.path.abspath(__file__))

    return os.path.abspath(os.path.dirname(__file__))

def __resolve_sdk():

    """Sets the SDK environment variables and imports the sd module"""

    sdk_root = pathlib.Path(os.environ.get('SD_SDK_ROOT', __get_run_path()))

    if not sdk_root.exists() or not sdk_root.is_dir():

        raise ImportError(f'{str(sdk_root)} is not a valid location. Make sure you set %SD_SDK_ROOT% appropriately in your environment.')

    # We'll fall back to this if we can't find sd.pyd in SD_SDK_ROOT

    sdk_module_folder = sdk_root / _WIN_SAMPLES_PATH

    # Try to find the sd.pyd file in sdk_root

    try_sd_pyd = sdk_root / 'sd.pyd'

    if try_sd_pyd.exists() and try_sd_pyd.is_file():

        logging.debug(f"Found 'sd.pyd' in {str(sdk_root)}")

        sdk_module_folder = sdk_root

    else:

        logging.debug(f"Failed to find 'sd.pyd' in {str(sdk_root)}. Using {str(sdk_module_folder)} instead.")

    sdk_module = sdk_module_folder / 'sd.pyd'

    sdk_config = sdk_module_folder / 'sd.config'

    if not sdk_module.exists() or not sdk_module.is_file():

        raise ImportError(f"Failed to find 'sd.pyd' in {str(sdk_module_folder)}! Make sure you set %SD_SDK_ROOT% appropriately in your environment.")

    # Set the SDK environment variables BEFORE attempting to import from sd

    # Note that the trailing separator is required for the SDK to work

    os.environ['SD_MODULE_PATH'] = str(sdk_module.parent) + '\\'

    os.environ['SD_CONFIG_PATH'] = str(sdk_config)

    os.environ["PATH"] += os.pathsep + str(sdk_module.parent)

    module_name = 'sd'

    sys.path.append(str(sdk_module.parent))

    spec = importlib.util.find_spec(module_name)

    if spec is not None:

        module = importlib.util.module_from_spec(spec)

        sys.modules[module_name] = module

        spec.loader.exec_module(module)

    else:

        raise ImportError(f"Failed to find module {module_name}")

__resolve_sdk()

import sd

It finds the sd.pyd file dynamically and sets everything up properly, but fails to load the .dll for some reason when turned into an .exe without the path "baked into" the executable via PYTHONPATH. 

Is there a way I can make this work with pyInstaller's heavily customized import mechanism?

Thanks,

Mark

[Mark Melvin]

Arg! I found os.add_dll_directory(path) but this still doesn't work with pyInstaller. My ModuleSpec says I have an ExtensionFileLoader instance to load my extension module, but it is failing for some reason, even though it is pointing to the exact .pyd file! Some weird interaction with the way pyInstaller bootloaded the interpreter I am guessing. Hmmm....

[Brenainn Woodsend]

Does this PYD file have dependencies – be them other PYDs, DLLs or Python libraries? This should be as simple as adding the parent directory to sys.path and running import sd so if there’s no dependencies at play here, I’d be surprised if you hadn’t got that working. The fact that you’re getting a DLL load failed error instead of a ModuleNotFoundError would also imply that your PYD is findable but not loadable.

I have tried manually setting the PATH and PYTHONPATH in the console and even copying the .pyd file to be alongside the .exe itself, but nothing works. Now the weird part. If I re-run the pyInstaller build with my PYTHONPATH set to the location where my .pyd file resides, it creates an executable that does work. So it appears that pyInstaller has its own notion of a PYTHONPATH that can be augmented at build time but not runtime??

This also suggests the same. I think PyInstaller sees that import sd, includes the PYD then runs it through its binary dependency analysis where it discovers whatever DLLs this PYD is linked against and collects them.

P.S. That __get_run_path() path function is redundant. You should use os.path.abspath(os.path.dirname(__file__)) unconditionally – adjusting any --add-data arguments so that the files are put where the code now expects them to be.

[Mark Melvin]

Yes, the .pyd depends on a .DLL. And normally adding the parent folder to sys.path is all I need to do, but once it's run through pyInstaller, this no longer works. Perhaps the issue is that the .pyd is being included in my .exe when I don't actually want it to be? 

I may just stuff everything in the .exe anyway in the end if I can't figure this out. Thanks for getting back to me.