[PATCH 2 of 3] packaging: build the MSI installer for py2exe on python3

[Matt Harbison]

# HG changeset patch
# User Matt Harbison <matt_h...@yahoo.com>
# Date 1651604131 14400
# Tue May 03 14:55:31 2022 -0400
# Node ID 69131fbd8995b7107fa3e0f6f2530400091f984e
# Parent 7e8efd53e69cb89c4d0f52aeab43a645796433e2
# EXP-Topic windows-py3-packaging
packaging: build the MSI installer for py2exe on python3

This is the point at which the py3 MSI equivalent to the winbuild script can
finally be built. Aside from some upstream issues (like pager needing to be
disabled) and likely str vs bytes issues, one of the few known issues is that
the curses module isn't included (because py2exe isn't seeing it for some
reason). That can be worked out later. The winbuild script also adds the hg
version as part of the MSI comment. We don't fetch it here, and assume it is
the same as the thg version because Windows always bundles the same hg and thg
version. The winbuild script also includes the hash of the shell extension as a
property. I have no idea where that shows up, and would like to vendor the
shell extension, so this script doesn't do that either.

It's a bit unfortunate that a couple of WiX files need to be copied, but I'd
rather not try to conditionalize them internally based on py2 vs py3. I'd also
like to move them to `contrib/packaging/wix` to mimic the Mercurial structure,
but that requires copying/moving the *.wxi files too, and that might get messy.
That can be handled in the future. By copying the VC runtime DLL instead of
using (the frowned upon) merge modules, I was able to build this both with VS
2017 and 2019 (which have different merge module files).

Additional future work in this area would be to vendor the shell extension, and
generate the *.wxs file based on a staging directory to make it easier to update
to new versions of python or include more extensions. But when I installed a
py3 Mercurial over a py2 Mercurial, it didn't clean up the py2 files. Since the
functions to generate the file were borrowed from Mercurial, I assume we'd have
the same problem here, so that needs some investigation first.

diff --git a/contrib/packaging/thgpackaging/py2exe.py b/contrib/packaging/thgpackaging/py2exe.py
--- a/contrib/packaging/thgpackaging/py2exe.py
+++ b/contrib/packaging/thgpackaging/py2exe.py
@@ -19,6 +19,7 @@
from .util import (
extract_tar_to_directory,
extract_zip_to_directory,
+ find_vc_runtime_dll,
process_install_rules,
python_exe_info,
SourceDirs,
@@ -397,6 +398,10 @@
for f in ("libcrypto-1_1.dll", "libssl-1_1.dll"):
shutil.move(dist_dir / "lib" / f, dist_dir)

+ # Add vcruntimeXXX.dll next to executable.
+ vc_runtime_dll = find_vc_runtime_dll(x64=vc_x64)
+ shutil.copy(vc_runtime_dll, dist_dir / vc_runtime_dll.name)
+

def stage_install(
source_dir: pathlib.Path, staging_dir: pathlib.Path, lower_case=False
diff --git a/contrib/packaging/thgpackaging/wix.py b/contrib/packaging/thgpackaging/wix.py
--- a/contrib/packaging/thgpackaging/wix.py
+++ b/contrib/packaging/thgpackaging/wix.py
@@ -330,6 +330,20 @@
# EXTRA_INSTALL_RULES, source_dirs.original, staging_dir
# )

+ # TODO: generate these in the staging area
+ dist_dir = source_dirs.thg / 'dist'
+
+ # kdiff3 needs qt.conf to find platforms and imageformat plugins
+ with open(dist_dir / 'lib' / 'qt.conf', 'w') as fp:
+ fp.write('[Paths]\nPlugins = ..\n')
+
+ # workaround for QTBUG 57687
+ # TODO: See if this can be dropped once we're past 5.9 on Windows with py2
+ shutil.copy(
+ source_dirs.winbuild / "contrib" / "spawn.cmd",
+ dist_dir / 'lib' / 'spawn.cmd'
+ )
+
# And remove some files we don't want.
# for f in STAGING_REMOVE_FILES:
# p = staging_dir / f
@@ -337,12 +351,37 @@
# print('removing %s' % p)
# p.unlink()

+ wix_dir = source_dirs.thg / 'win32' / 'wix'
+
+ # TODO: map these to the staging area
+ wxs_entries = {
+ wix_dir / 'diff-scripts.wxs':
+ source_dirs.winbuild / 'contrib' / 'diff-scripts',
+ wix_dir / 'dist-py3.wxs': source_dirs.thg / 'dist',
+ wix_dir / 'icons.wxs': source_dirs.thg / 'dist' / 'icons',
+ wix_dir / 'thg-locale.wxs': source_dirs.thg / 'locale',
+ source_dirs.shellext / 'wix' / 'cmenu-i18n.wxs':
+ source_dirs.thg / 'win32',
+ source_dirs.shellext / 'wix' / 'thgshell.wxs': source_dirs.shellext,
+ wix_dir / 'help.wxs': source_dirs.hg / 'mercurial' / 'helptext',
+ wix_dir / 'templates.wxs': source_dirs.hg / 'mercurial' / 'templates',
+ wix_dir / 'locale.wxs': source_dirs.hg / 'mercurial' / 'locale',
+ wix_dir / 'i18n.wxs': source_dirs.hg / 'i18n',
+ wix_dir / 'doc.wxs': source_dirs.hg / 'doc',
+ wix_dir / 'contrib.wxs': source_dirs.hg / 'contrib',
+ }
+
+ extra_wxs = dict(extra_wxs if extra_wxs else {})
+ for wxs, source_dir in wxs_entries.items():
+ extra_wxs[str(wxs)] = str(source_dir)
+
return run_wix_packaging(
- source_dir,
+ source_dirs,
build_dir,
staging_dir,
arch,
version=version,
+ productid=productid,
msi_name=msi_name,
extra_wxs=extra_wxs,
extra_features=extra_features,
@@ -428,11 +467,12 @@


def run_wix_packaging(
- source_dir: pathlib.Path,
+ source_dirs: SourceDirs,
build_dir: pathlib.Path,
staging_dir: pathlib.Path,
arch: str,
version: str,
+ productid: str,
msi_name: typing.Optional[str] = "tortoisehg",
suffix: str = "",
extra_wxs: typing.Optional[typing.Dict[str, str]] = None,
@@ -448,13 +488,16 @@
specified.
"""

- orig_version = version or read_version_py(source_dir)
+ orig_version = version or read_version_py(source_dirs.thg)
+ # TODO: fix this hack that's also done in the shellext build
+ orig_version = orig_version.split('+', 1)[0]
version = normalize_windows_version(orig_version)
print('using version string: %s' % version)
if version != orig_version:
print('(normalized from: %s)' % orig_version)

if signing_info:
+ # TODO: sign other executables
sign_with_signtool(
staging_dir / "hg.exe",
"%s %s" % (signing_info["name"], version),
@@ -464,7 +507,7 @@
timestamp_url=signing_info["timestamp_url"],
)

- wix_dir = source_dir / 'contrib' / 'packaging' / 'wix'
+ wix_dir = source_dirs.thg / 'win32' / 'wix'

wix_pkg, wix_entry = download_entry('wix', build_dir)
wix_path = build_dir / ('wix-%s' % wix_entry['version'])
@@ -472,37 +515,57 @@
if not wix_path.exists():
extract_zip_to_directory(wix_pkg, wix_path)

- source_build_rel = pathlib.Path(os.path.relpath(source_dir, build_dir))
+ # TODO: drop this path hacking when the following relative path errors are
+ # fixed (after py2 is dropped):
+ # dist-py3.wxs(106) : error LGHT0103 : The system cannot find the file '..\contrib\kdiff3x64.exe'.
+ # tortoisehg-py3.wxs(100) : error LGHT0103 : The system cannot find the file '..\extension-versions.txt'.
+ # tortoisehg-py3.wxs(116) : error LGHT0103 : The system cannot find the file '..\contrib\Pageant-x64.exe'.
+ # tortoisehg-py3.wxs(139) : error LGHT0103 : The system cannot find the file '..\misc\hgbook.pdf'.
+ # dist-py3.wxs(100) : error LGHT0103 : The system cannot find the file '..\contrib\kdiff3x64.exe'.
+ # dist-py3.wxs(110) : error LGHT0103 : The system cannot find the file '..\contrib\TortoisePlink-x64.exe'.
+ # thgshell.wxs(72) : error LGHT0103 : The system cannot find the file '../contrib/TortoiseOverlays\TortoiseOverlays-1.1.3.21564-x64.msm'.
+ # thgshell.wxs(76) : error LGHT0103 : The system cannot find the file '../contrib/TortoiseOverlays\TortoiseOverlays-1.1.3.21564-win32.msm'.
+ build_dir = source_dirs.thg
+
+ source_build_rel = pathlib.Path(os.path.relpath(source_dirs.thg, build_dir))

defines = {'Platform': arch}

# Derive a .wxs file with the staged files.
- manifest_wxs = build_dir / 'stage.wxs'
- with manifest_wxs.open('w', encoding='utf-8') as fh:
- fh.write(make_files_xml(staging_dir, is_x64=arch == 'x64'))
-
- run_candle(wix_path, build_dir, manifest_wxs, staging_dir, defines=defines)
+ # manifest_wxs = build_dir / 'stage.wxs'
+ # with manifest_wxs.open('w', encoding='utf-8') as fh:
+ # fh.write(make_files_xml(staging_dir, is_x64=arch == 'x64'))
+ #
+ # run_candle(wix_path, build_dir, manifest_wxs, staging_dir, defines=defines)

for source, rel_path in sorted((extra_wxs or {}).items()):
run_candle(wix_path, build_dir, source, rel_path, defines=defines)

- source = wix_dir / 'mercurial.wxs'
+ source = wix_dir / 'tortoisehg-py3.wxs'
+
+ # TODO: get the actual hg value
+ hgversion = version
+
defines['Version'] = version
- defines['Comments'] = 'Installs TortoiseHg version %s' % version
+ defines['Comments'] = 'Installs TortoiseHg %s, Mercurial %s on %s' % (
+ version, hgversion, arch,
+ )

defines["PythonVersion"] = "3"
+ defines['ProductId'] = productid
+ defines['ShellextRepoFolder'] = str(source_dirs.shellext)

- if (staging_dir / "lib").exists():
- defines["MercurialHasLib"] = "1"
+ # if (staging_dir / "lib").exists():
+ # defines["TortoiseHgHasLib"] = "1"

if extra_features:
assert all(';' not in f for f in extra_features)
- defines['MercurialExtraFeatures'] = ';'.join(extra_features)
+ defines['TortoiseHgExtraFeatures'] = ';'.join(extra_features)

run_candle(wix_path, build_dir, source, source_build_rel, defines=defines)

msi_path = (
- source_dir
+ source_dirs.original
/ 'dist'
/ ('%s-%s-%s%s.msi' % (msi_name, orig_version, arch, suffix))
)
@@ -525,12 +588,14 @@

args.extend(
[
- str(build_dir / 'stage.wixobj'),
- str(build_dir / 'mercurial.wixobj'),
+ # str(build_dir / 'stage.wixobj'),
+ str(build_dir / 'tortoisehg-py3.wixobj'),
]
)

- subprocess.run(args, cwd=str(source_dir), check=True)
+ # TODO: drop this `.parent` path altering when relative paths are cleaned
+ # up in the *.wxs files
+ subprocess.run(args, cwd=str(source_dirs.thg.parent), check=True)

print('%s created' % msi_path)

diff --git a/win32/wix/dist.wxs b/win32/wix/dist-py3.wxs
copy from win32/wix/dist.wxs
copy to win32/wix/dist-py3.wxs
--- a/win32/wix/dist.wxs
+++ b/win32/wix/dist-py3.wxs
@@ -9,15 +9,29 @@
<Component Id="distOutput" Guid="$(var.dist.guid)" Win64='$(var.IsX64)'>
<File Name="thg.exe" KeyPath="yes" />
<File Name="hg.exe" />
- <File Name="python27.dll" />
- <File Name="ssleay32.dll" />
- <File Name="libeay32.dll" />
+ <File Name="vcruntime140.dll" />
+ <File Name="python39.dll" />
+ <?if $(var.Platform) = "x64" ?>
+ <File Name="libcrypto-1_1-x64.dll" />
+ <File Name="libssl-1_1-x64.dll" />
+ <?else?>
+ <File Name="libcrypto-1_1.dll" />
+ <File Name="libssl-1_1.dll" />
+ <?endif?>
</Component>
<Directory Id="libdir" Name="lib" FileSource="$(var.SourceDir)/lib">
<Component Id="libOutput" Guid="$(var.lib.guid)" Win64='$(var.IsX64)'>
<File Name="library.zip" KeyPath="yes" />
- <File Name="pythoncom27.dll" />
- <File Name="pywintypes27.dll" />
+ <File Name="python3.dll" />
+ <File Name="pythoncom39.dll" />
+ <File Name="pywintypes39.dll" />
+ <File Name="libffi-7.dll" />
+ <?if $(var.Platform) = "x64" ?>
+ <File Name="libcrypto-1_1.dll" />
+ <File Name="libssl-1_1.dll" />
+ <?endif?>
+ <File Name="pygit2._libgit2.pyd" />
+ <File Name="pygit2._pygit2.pyd" />
<File Name="Qt5Core.dll" />
<File Name="Qt5Gui.dll" />
<File Name="Qt5Widgets.dll" />
@@ -26,7 +40,6 @@
<File Name="Qt5Svg.dll" />
<File Name="Qt5Xml.dll" />
<File Name="qt.conf" />
- <File Name="qscintilla2_qt5.dll" />
<File Name="sqlite3.dll" />
<File Name="mercurial.cext.base85.pyd" />
<File Name="mercurial.cext.bdiff.pyd" />
@@ -46,9 +59,10 @@
<File Name="PyQt5.QtNetwork.pyd" />
<File Name="PyQt5.QtSvg.pyd" />
<File Name="PyQt5.QtXml.pyd" />
- <File Name="bz2.pyd" />
<File Name="select.pyd" />
<File Name="PyQt5.sip.pyd" />
+ <File Name="tcl86t.dll" />
+ <File Name="tk86t.dll" />
<File Name="unicodedata.pyd" />
<File Name="win32api.pyd" />
<File Name="win32com.shell.shell.pyd" />
@@ -60,17 +74,28 @@
<File Name="win32process.pyd" />
<File Name="win32security.pyd" />
<File Name="win32trace.pyd" />
+ <File Name="_asyncio.pyd" />
+ <File Name="_bz2.pyd" />
+ <File Name="_cffi_backend.pyd" />
<File Name="_ctypes.pyd" />
- <File Name="_ctypes_test.pyd" />
- <File Name="_curses.pyd" />
+ <!-- TODO: enable this when py2exe recognizes the curses module -->
+ <!-- <File Name="_curses.pyd" -->
<File Name="_curses_panel.pyd" />
+ <File Name="_decimal.pyd" />
<File Name="_elementtree.pyd" />
<File Name="_hashlib.pyd" />
+ <File Name="_lzma.pyd" />
<File Name="_multiprocessing.pyd" />
+ <File Name="_overlapped.pyd" />
+ <File Name="_queue.pyd" />
<File Name="_socket.pyd" />
<File Name="_ssl.pyd" />
<File Name="_sqlite3.pyd" />
+ <File Name="_tkinter.pyd" />
+ <File Name="_uuid.pyd" />
<File Name="_win32sysloader.pyd" />
+ <File Name="cacert.pem" />
+ <File Name="git2.dll" />
<File Name="spawn.cmd" />
</Component>
<?if $(var.Platform) = "x64" ?>
diff --git a/win32/wix/guids.wxi b/win32/wix/guids.wxi
--- a/win32/wix/guids.wxi
+++ b/win32/wix/guids.wxi
@@ -32,8 +32,8 @@
<?define templates.static.guid = {2c807a50-27ce-46c4-b8e6-1cb0be1e7105} ?>

<!-- dist.wxs -->
- <?define dist.guid = {4f1c06dd-8141-4761-beac-f96311dce1e1} ?>
- <?define lib.guid = {ddf40719-de49-4211-bd58-6331b00d8cc6} ?>
+ <?define dist.guid = {e39b0f21-034e-45fa-9da5-802148c1a883} ?>
+ <?define lib.guid = {da5cb94e-dae4-45b2-a9b3-8138e97b5d71} ?>
<?define imageformats.guid = {12885995-61ed-468f-a964-3654d091138e} ?>
<?define platforms.guid = {4b2c42d4-fbb3-4449-b1df-31bf789e009e} ?>
<?define KDiff3EXE.guid = {075ECC11-1B44-48DB-B7FD-D3207BB801A4} ?>
diff --git a/win32/wix/tortoisehg.wxs b/win32/wix/tortoisehg-py3.wxs
copy from win32/wix/tortoisehg.wxs
copy to win32/wix/tortoisehg-py3.wxs
--- a/win32/wix/tortoisehg.wxs
+++ b/win32/wix/tortoisehg-py3.wxs
@@ -212,23 +212,6 @@
</Component>
</Directory>
</Directory>
-
- <?if $(var.Platform) = "x86" ?>
- <Merge Id='VCRedist' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\microsoft.vcxx.crt.x86_msm.msm'/>
- <Merge Id='VCRedistPolicy' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\policy.x.xx.microsoft.vcxx.crt.x86_msm.msm'/>
- <Merge Id='VCRedist2019' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\Microsoft_VC142_CRT_x86.msm'/>
- <?else?>
- <Merge Id='VCRedist' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\microsoft.vcxx.crt.x64_msm.msm'/>
- <Merge Id='VCRedistPolicy' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\policy.x.xx.microsoft.vcxx.crt.x64_msm.msm'/>
- <Merge Id='VCRedist2019' DiskId='1' Language='1033'
- SourceFile='$(var.VCRedistSrcDir)\Microsoft_VC142_CRT_x64.msm'/>
- <?endif?>
-
</Directory>

<Feature Id='Complete' Title='TortoiseHg'
@@ -255,15 +238,6 @@
<ComponentGroupRef Id='templatesFolder' />
<ComponentRef Id='Icons' />
</Feature>
- <Feature Id='VCRedist' Title='Visual C++ 9.0 Runtime'
- AllowAdvertise='no' Display='hidden' Level='1'>
- <MergeRef Id='VCRedist'/>
- <MergeRef Id='VCRedistPolicy' />
- </Feature>
- <Feature Id='VCRedist2019' Title='Visual C++ 2019 Runtime'
- AllowAdvertise='no' Display='hidden' Level='1'>
- <MergeRef Id='VCRedist2019'/>
- </Feature>
<!-- referencing of the shellext features with FeatureRef sorts the
shellext features in some unknown and not controllable ridiculous
order in the feature dialog, so we have to include them here to

[Matt Harbison]

On Tuesday, May 3, 2022 at 10:50:29 PM UTC-4 Matt Harbison wrote:

# HG changeset patch
# User Matt Harbison <matt_h...@yahoo.com>
# Date 1651604131 14400
# Tue May 03 14:55:31 2022 -0400
# Node ID 69131fbd8995b7107fa3e0f6f2530400091f984e
# Parent 7e8efd53e69cb89c4d0f52aeab43a645796433e2
# EXP-Topic windows-py3-packaging
packaging: build the MSI installer for py2exe on python3

@@ -337,12 +351,37 @@
# print('removing %s' % p)
# p.unlink()

+ wix_dir = source_dirs.thg / 'win32' / 'wix'
+
+ # TODO: map these to the staging area
+ wxs_entries = {
+ wix_dir / 'diff-scripts.wxs':

This is the wrong path for stable, since the diff script vendoring landed on default.  The value part below does work for stable, so I can either respin this for stable or apply these to default.

I've been hacking on default because some hg changes needed to work with py2exe on py3 are on default, but hg-default dropped pycompat.bytesio, and the fix for that (7fe8827b5025) is on thg-default.

[Yuya Nishihara]

On Wed, 4 May 2022 08:57:32 -0700 (PDT), 'Matt Harbison' via TortoiseHg Developers wrote:
> On Tuesday, May 3, 2022 at 10:50:29 PM UTC-4 Matt Harbison wrote:
>
> > # HG changeset patch
> > # User Matt Harbison <matt_h...@yahoo.com>
> > # Date 1651604131 14400
> > # Tue May 03 14:55:31 2022 -0400
> > # Node ID 69131fbd8995b7107fa3e0f6f2530400091f984e
> > # Parent 7e8efd53e69cb89c4d0f52aeab43a645796433e2
> > # EXP-Topic windows-py3-packaging
> > packaging: build the MSI installer for py2exe on python3
> >
> >
> > @@ -337,12 +351,37 @@
> > # print('removing %s' % p)
> > # p.unlink()
> >
> > + wix_dir = source_dirs.thg / 'win32' / 'wix'
> > +
> > + # TODO: map these to the staging area
> > + wxs_entries = {
> > + wix_dir / 'diff-scripts.wxs':
> >
>
> This is the wrong path for stable, since the diff script vendoring landed
> on default. The value part below does work for stable, so I can either
> respin this for stable or apply these to default.

Grafted the diff-scripts patch to stable. Let me know if something got broken.

Queued the series for stable, thanks.

[Matt Harbison]

Thanks.  I was able to build the installer for x86 and x64 on stable, so I think everything is OK.