Using the Windows "embedded" distribution of Python

Paul Moore

unread,

Sep 28, 2016, 10:39:08 AM9/28/16

to

This is probably more of a Windows question than a Python question, but as it's related to embedding Python, I thought I'd try here anyway.

I'm writing some command line applications in Python, and I want to bundle them into a standalone form using the new "embedding" distribution of Python. The basic process is easy enough - write a small exe that sets up argv, and then call Py_Main. Furthermore, as I'm *only* using Py_Main, I can use the restricted API, and make my code binary compatible with any Python 3 version.

So far so good. I have my exe, and my application code. I create a directory for my app, put the exe/app code in there, and then unzip the embedded distribution in there. And everything works. Yay!

However, I'm expecting my users to put my application directory on their PATH (as these are command line utilities). And I don't really want my private copy of the Python DLLs to be exposed on the user's PATH (I don't *know* that it'll interfere with their actual Python installation, but I'd prefer not to take risks). And this is when my problems start. Ideally, I'd put the embedded Python distribution in a subdirectory (say "embed") of my application directory. But the standard loader won't find python3.dll from there. So I have to do something a bit better.

As I'm only using Py_Main, I thought I'd be OK to dynamically load it - LoadLibrary on the DLL, then GetProcAddress. No big deal. But I need to tell LoadLibrary to get the DLL from the "embed" subdirectory. I suppose I could add that directory to PATH locally to my program, but that seems clumsy. So I thought I'd try SetDllDirectory. That works for python36.dll, but if I load python3.dll, it can't find Py_Main - the export shows as "(forwarded to python36.Py_Main)", maybe the forwarding doesn't handle SetDllDirectory?

So, basically what I'm asking:

* If I want to put my application on PATH, am I stuck with having the embedded distribution in the same directory, and also on PATH?
* If I can put the embedded distribution in a subdirectory, can that be made to work with python3.dll, or will I have to use python36.dll?

None of this is an issue with the most likely use of the embedded distribution (GUI apps, or server apps, both of which are likely to be run by absolute path, and so don't need to be on PATH myself). But I'd really like to be able to promote the embedded distribution as an alternative to tools like py2exe or cx_Freeze, so it would be good to know if a solution is possible (hmm, how come py2exe, and tools like Mercurial, which AFIK use it, don't have this issue too?)

Paul

eryk sun

unread,

Sep 28, 2016, 4:50:54 PM9/28/16

to

On Wed, Sep 28, 2016 at 2:35 PM, Paul Moore <p.f....@gmail.com> wrote:
> So I thought I'd try SetDllDirectory. That works for python36.dll, but if I load
> python3.dll, it can't find Py_Main - the export shows as "(forwarded to
> python36.Py_Main)", maybe the forwarding doesn't handle SetDllDirectory?

It works for me. Are you calling SetDllDirectory with the
fully-qualified path? If not it's relative to the working directory,
which isn't necessarily (generally is not) the application directory,
in which case delay-loading python36.dll will fail. You can create the
fully-qualified path from the application path, i.e.
GetModuleFileNameW(NULL, ...).

That said, I prefer using LoadLibraryExW(absolute_path_to_python3,
NULL, LOAD_WITH_ALTERED_SEARCH_PATH). The alternate search substitutes
the DLL's directory for the application directory when loading
dependent DLLs, which allows loading python36.dll and vcruntime140.dll
without having to modify the DLL search path of the entire process.

Paul Moore

unread,

Sep 29, 2016, 4:35:57 AM9/29/16

to

On Wednesday, 28 September 2016 21:50:54 UTC+1, eryk sun wrote:
> On Wed, Sep 28, 2016 at 2:35 PM, Paul Moore <p.f....@gmail.com> wrote:
> > So I thought I'd try SetDllDirectory. That works for python36.dll, but if I load
> > python3.dll, it can't find Py_Main - the export shows as "(forwarded to
> > python36.Py_Main)", maybe the forwarding doesn't handle SetDllDirectory?
>
> It works for me. Are you calling SetDllDirectory with the
> fully-qualified path? If not it's relative to the working directory,
> which isn't necessarily (generally is not) the application directory,
> in which case delay-loading python36.dll will fail. You can create the
> fully-qualified path from the application path, i.e.
> GetModuleFileNameW(NULL, ...).

That might be the issue. I was using a relative directory (I was being lazy for a quick test) but I was at the time in the right directory for the relative directory name to work. But I'll do a proper test today.

> That said, I prefer using LoadLibraryExW(absolute_path_to_python3,
> NULL, LOAD_WITH_ALTERED_SEARCH_PATH). The alternate search substitutes
> the DLL's directory for the application directory when loading
> dependent DLLs, which allows loading python36.dll and vcruntime140.dll
> without having to modify the DLL search path of the entire process.

That does indeed sound like a better idea. I had read the docs for LoadLibraryEx, but I must admit I got very muddled from the various options, and ended up going back to LoadLibrary because it seemed simpler :-(

Thanks for the suggestions,
Paul

PS It's a shame there's no way to put the embedded distribution in a subdirectory *without* needing to use dynamic loading, but I guess that's basically an OS limitation.

eryk sun

unread,

Sep 29, 2016, 5:39:10 AM9/29/16

to

On Thu, Sep 29, 2016 at 8:35 AM, Paul Moore <p.f....@gmail.com> wrote:
> PS It's a shame there's no way to put the embedded distribution in a subdirectory
> *without* needing to use dynamic loading, but I guess that's basically an OS limitation.

There are ways to do this. The simplest way is to use a subdirectory
with same name as the executable plus ".local". For example, for
"app.exe" create a directory named "app.exe.local".

Paul Moore

unread,

Sep 29, 2016, 6:41:32 AM9/29/16

to

Oh, wow. Now you mention it, I recall that convention (from somewhere). I'll investigate that option (although it may not suit my use case, as I want multiple exes in the one "main" directory sharing a single "local" Python runtime).

Many thanks,
Paul

eryk sun

unread,

Sep 29, 2016, 7:56:28 AM9/29/16

to

In that case you can use an application manifest with a dependent
assembly. Say embedded Python 3.6 is in the "py3embed" subdirectory.
Add the following manifest file to that directory:

py3embed.manifest:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<assemblyIdentity name="py3embed"
version="3.6.111.1013"
type="win32"
processorArchitecture="amd64" />
<file name="python3.dll" />
<file name="python36.dll" />
<file name="vcruntime140.dll" />
</assembly>

Add this assembly as a dependency in the application manifest, either
embedded in the executable (resource #1) or as a separate file named
the same as the executable plus ".manifest", e.g. "app.exe.manifest".
You can start with the manifest that's used by python.exe, from
PC\python.manifest.

<dependency>
<dependentAssembly>
<assemblyIdentity name="py3embed"
version="3.6.111.1013"
type="win32"
processorArchitecture="amd64" />
</dependentAssembly>
</dependency>

Paul Moore

unread,

Sep 29, 2016, 10:05:11 AM9/29/16

to

On Thursday, 29 September 2016 12:56:28 UTC+1, eryk sun wrote:
>> Oh, wow. Now you mention it, I recall that convention (from somewhere). >> I'll investigate that option (although it may not suit my use case, as
>> I want multiple exes in the one "main" directory sharing a single
>> "local" Python runtime).
>
> In that case you can use an application manifest with a dependent
> assembly. Say embedded Python 3.6 is in the "py3embed" subdirectory.
> Add the following manifest file to that directory:
>
> py3embed.manifest:
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
> <assemblyIdentity name="py3embed"
> version="3.6.111.1013"
> type="win32"
> processorArchitecture="amd64" />
> <file name="python3.dll" />
> <file name="python36.dll" />
> <file name="vcruntime140.dll" />
> </assembly>

Cool. So this is all the SxS stuff that I never really understood, right? :-) I guess I can assume from this:

1. The filename py3embed.manifest isn't relevant (you explain file naming below).
2. assemblyIdentity name="py3embed" is what says to look in a subdirectory "py3embed" that's located next to the executable.
3. The "version" in the manifest doesn't really matter much.
4. I only need to name python3.dll, python36.dll and vcruntime140.dll as these are the only DLLs loaded statically?

> Add this assembly as a dependency in the application manifest, either
> embedded in the executable (resource #1) or as a separate file named
> the same as the executable plus ".manifest", e.g. "app.exe.manifest".

Using a separate file seems easiest (and certainly the way to go for testing) but I'm not sure how I'd embed the manifest using command line tools. I'm leveraging distutils to build my exe at the moment:

from distutils.ccompiler import new_compiler
import distutils.sysconfig
import sys
import os

cc = new_compiler()
exe = 'myapp'
cc.add_include_dir(distutils.sysconfig.get_python_inc())
cc.add_library_dir(os.path.join(sys.base_exec_prefix, 'libs'))
objs = cc.compile(['myapp.c'])
cc.link_executable(objs, exe)

But the ccompiler interface doesn't have anything to add in a resource or manifest, so I think I'm going to need to switch to using the command line directly for this.

> You can start with the manifest that's used by python.exe, from
> PC\python.manifest.
>
> <dependency>
> <dependentAssembly>
> <assemblyIdentity name="py3embed"
> version="3.6.111.1013"
> type="win32"
> processorArchitecture="amd64" />
> </dependentAssembly>
> </dependency>

Thanks for your help on this. My C skills are *very* rusty (going much beyond "cl /c foo.c" sends me to the manuals these days) so I appreciate all the help.

I'm off now to do a whole load of experimenting - you've given me a lot to work with :-)

Paul.

PS Is there any readable documentation on the SxS stuff anywhere, written for C programmers? Microsoft's docs have always seemed to me to assume I know a bunch of .NET concepts that I really don't know much about...

Paul Moore

unread,

Sep 30, 2016, 7:05:59 AM9/30/16

to

On Thursday, 29 September 2016 12:56:28 UTC+1, eryk sun wrote:

OK, I thought I understood this, but it's not quite working.

I have the following directory structure:

py3embed
py3embed.manifest
... an unpacked Python 3.6b0 embedded distribution
ssh.exe
ssh.exe.manifest

(ssh.exe is my embedded Python program)

py3embed.manifest is

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<assemblyIdentity name="py3embed"
version="3.6.111.1013"
type="win32"
processorArchitecture="amd64" />
<file name="python3.dll" />
<file name="python36.dll" />
<file name="vcruntime140.dll" />
</assembly>

ssh.exe.manifest is:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">

</assembly>

When I run ssh.exe, it fails with the message "The program cannot start because python3.dll is missing from your computer". I tried running it with sxstrace active, but the resulting log file is empty.

I'm not sure where to go next debugging this. Do you have any suggestions?

Paul

eryk sun

unread,

Sep 30, 2016, 7:50:45 AM9/30/16

to

On Fri, Sep 30, 2016 at 11:02 AM, Paul Moore <p.f....@gmail.com> wrote:
> When I run ssh.exe, it fails with the message "The program cannot start because
> python3.dll is missing from your computer". I tried running it with sxstrace active,
> but the resulting log file is empty.

A manifest embedded in "ssh.exe" takes precedence over
"ssh.exe.manifest". Check for an embedded manifest:

mt -inputresource:ssh.exe -out:con >nul

Also, manifests get cached (probably by SuperFetch), so when you make
a change you generally need to recompile or change the filename.

Paul Moore

unread,

Sep 30, 2016, 9:15:26 AM9/30/16

to

On Friday, 30 September 2016 12:50:45 UTC+1, eryk sun wrote:
> On Fri, Sep 30, 2016 at 11:02 AM, Paul Moore <p.f....@gmail.com> wrote:
> > When I run ssh.exe, it fails with the message "The program cannot start because
> > python3.dll is missing from your computer". I tried running it with sxstrace active,
> > but the resulting log file is empty.
>
> A manifest embedded in "ssh.exe" takes precedence over
> "ssh.exe.manifest". Check for an embedded manifest:
>
> mt -inputresource:ssh.exe -out:con >nul

I didn't think there was one, but when I looked, there is. I must have added it in my experimentation. Sadly, it's the same as the external one, so that shouldn't be a problem.

> Also, manifests get cached (probably by SuperFetch), so when you make
> a change you generally need to recompile or change the filename.

Ah, looks like that was the issue. Renaming the file fixed it. (As did just touching the file to update the timestamp). Caches are great, I guess, but they sure make debugging an adventure!

I think I now have a working solution. And I've *definitely* learned a huge amount.

Thanks again for the help.
Paul