Re: Bug#934928: win32-loader FTBFS on stable - any idea ?

[Simon McVittie]

On Fri, 23 Aug 2019 at 18:35:59 +0200, Didier 'OdyX' Raboud wrote:
> I have uploaded win32-loader to buster to fix the out-of-sync archive keys,
> but it has now repeatedly failed to build from source on the buster buildds:
>
> https://buildd.debian.org/status/logs.php?pkg=win32-loader&ver=0.9.4%2Bdeb10u1
>
> I can't reproduce this in a local buster schroot, so I'm slightly at loss.

I can reproduce this by trying to rebuild 0.9.4 in sbuild (a buster
schroot created with sbuild-createchroot, hosted on a buster VM, using
vectis[1]) if that's any help? Presumably there is some difference
between your buster schroot and the one sbuild would use. See below
for a full package list.

0.9.4 was originally compiled in January, so a lot has changed since then.
In particular the mingw gcc 7 has been replaced by gcc 8.

> Unable to convert processed string "نوشتن ممکن نیست: " to codepage 1256

That string appears to be part of the Farsi (fa) translation of nsis,
found in "Contrib/Language files/Farsi.nlf" in nsis_3.04-1. It is indeed
not possible to convert it to Windows codepage 1256:

$ python3
>>> "نوشتن ممکن نیست".encode('cp1256')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/encodings/cp1256.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u06cc' in position 12: character maps to <undefined>
$ unicode U+06CC
U+06CC ARABIC LETTER FARSI YEH

(I don't know whether converting this string to CP1256 is an appropriate
thing to be doing.)

If that's the problem, maybe it would be possible to work around this by
adjusting or disabling the Farsi translation, or by replacing the use of
CP1256 with UTF-16 or something?

smcv

[1] https://salsa.debian.org/smcv/vectis

----

Packages in my sbuild chroot when I reproduced this build failure:

adduser_3.118
adwaita-icon-theme_3.30.1-1
apt_1.8.2
autoconf_2.69-11
automake_1:1.16.1-4
autopoint_0.19.8.1-9
autotools-dev_20180224.1
base-files_10.3
base-passwd_3.5.46
bash_5.0-4
binutils_2.31.1-16
binutils-common_2.31.1-16
binutils-mingw-w64-i686_2.31.1-11+8.3
binutils-mingw-w64-x86-64_2.31.1-11+8.3
binutils-x86-64-linux-gnu_2.31.1-16
bsdmainutils_11.1.2+b1
bsdutils_1:2.33.1-0.1
build-essential_12.6
bzip2_1.0.6-9.1
ca-certificates_20190110
coreutils_8.30-3
cpio-win32_2.12+dfsg-9
cpp_4:8.3.0-1
cpp-8_8.3.0-6
dash_0.5.10.2-5
dbus_1.12.16-1
dbus-user-session_1.12.16-1
dconf-gsettings-backend_0.30.1-2
dconf-service_0.30.1-2
debconf_1.5.71
debhelper_12.1.1
debian-archive-keyring_2019.1
debianutils_4.8.6.1
dh-autoreconf_19
dh-strip-nondeterminism_1.1.2-1
diffutils_1:3.7-3
dmsetup_2:1.02.155-3
dpkg_1.19.7
dpkg-dev_1.19.7
dwz_0.12-3
e2fsprogs_1.44.5-1
fakeroot_1.23-1
fdisk_2.33.1-0.1
file_1:5.35-4
findutils_4.6.0+git+20190209-2
fontconfig_2.13.1-2
fontconfig-config_2.13.1-2
fonts-dejavu-core_2.37-1
g++_4:8.3.0-1
g++-8_8.3.0-6
g++-mingw-w64_8.3.0-6+21.3~deb10u1
g++-mingw-w64-i686_8.3.0-6+21.3~deb10u1
g++-mingw-w64-x86-64_8.3.0-6+21.3~deb10u1
gawk_1:4.2.1+dfsg-1
gcc_4:8.3.0-1
gcc-8_8.3.0-6
gcc-8-base_8.3.0-6
gcc-mingw-w64_8.3.0-6+21.3~deb10u1
gcc-mingw-w64-base_8.3.0-6+21.3~deb10u1
gcc-mingw-w64-i686_8.3.0-6+21.3~deb10u1
gcc-mingw-w64-x86-64_8.3.0-6+21.3~deb10u1
gettext_0.19.8.1-9
gettext-base_0.19.8.1-9
glib-networking_2.58.0-2
glib-networking-common_2.58.0-2
glib-networking-services_2.58.0-2
gpgv_2.2.12-1
gpgv-win32_2.2.12-1
grep_3.3-1
groff-base_1.22.4-3
grub-common_2.02+dfsg1-20
grub-pc-bin_2.02+dfsg1-20
gsettings-desktop-schemas_3.28.1-1
gtk-update-icon-cache_3.24.5-1
gzip_1.9-3
gzip-win32_1.9-3
hicolor-icon-theme_0.17-2
hostname_3.21
icoutils_0.32.3-2.1
imagemagick_8:6.9.10.23+dfsg-2.1
imagemagick-6-common_8:6.9.10.23+dfsg-2.1
imagemagick-6.q16_8:6.9.10.23+dfsg-2.1
init-system-helpers_1.56+nmu1
intltool-debian_0.35.0+20060710.5
ipxe_1.0.0+git-20190125.36a4c85-1
libacl1_2.2.53-4
libapparmor1_2.13.2-10
libapt-pkg5.0_1.8.2
libarchive-zip-perl_1.64-1
libargon2-1_0~20171227-0.2
libasan5_8.3.0-6
libatk-bridge2.0-0_2.30.0-5
libatk1.0-0_2.30.0-2
libatk1.0-data_2.30.0-2
libatomic1_8.3.0-6
libatspi2.0-0_2.30.0-7
libattr1_1:2.4.48-4
libaudit-common_1:2.8.4-3
libaudit1_1:2.8.4-3
libavahi-client3_0.7-4+b1
libavahi-common-data_0.7-4+b1
libavahi-common3_0.7-4+b1
libbinutils_2.31.1-16
libblkid1_2.33.1-0.1
libbsd0_0.9.1-2
libbz2-1.0_1.0.6-9.1
libc-bin_2.28-10
libc-dev-bin_2.28-10
libc-l10n_2.28-10
libc6_2.28-10
libc6-dev_2.28-10
libcairo-gobject2_1.16.0-4
libcairo2_1.16.0-4
libcap-ng0_0.7.9-2
libcap2_1:2.25-2
libcc1-0_8.3.0-6
libcolord2_1.4.3-4
libcom-err2_1.44.5-1
libcroco3_0.6.12-3
libcryptsetup12_2:2.1.0-5
libcups2_2.2.10-6
libdatrie1_0.2.12-2
libdb5.3_5.3.28+dfsg1-0.5
libdbus-1-3_1.12.16-1
libdconf1_0.30.1-2
libde265-0_1.0.3-1+b1
libdebconfclient0_0.249
libdevmapper1.02.1_2:1.02.155-3
libdpkg-perl_1.19.7
libefiboot1_37-2
libefivar1_37-2
libelf1_0.176-1.1
libencode-locale-perl_1.05-1
libepoxy0_1.5.3-0.1
libexpat1_2.2.6-2
libext2fs2_1.44.5-1
libfakeroot_1.23-1
libfdisk1_2.33.1-0.1
libffi6_3.2.1-9
libfftw3-double3_3.3.8-2
libfile-listing-perl_6.04-1
libfile-stripnondeterminism-perl_1.1.2-1
libfontconfig1_2.13.1-2
libfreetype6_2.9.1-3
libfribidi0_1.0.5-3.1
libfuse2_2.9.9-1
libgcc-8-dev_8.3.0-6
libgcc1_1:8.3.0-6
libgcrypt-mingw-w64-dev_1.8.4-5
libgcrypt20_1.8.4-5
libgdbm-compat4_1.18.1-4
libgdbm6_1.18.1-4
libgdk-pixbuf2.0-0_2.38.1+dfsg-1
libgdk-pixbuf2.0-common_2.38.1+dfsg-1
libglib2.0-0_2.58.3-2
libgmp10_2:6.1.2+dfsg-4
libgnutls30_3.6.7-4
libgomp1_8.3.0-6
libgpg-error-mingw-w64-dev_1.35-1
libgpg-error0_1.35-1
libgpm2_1.20.7-5
libgraphite2-3_1.3.13-7
libgssapi-krb5-2_1.17-3
libgtk-3-0_3.24.5-1
libgtk-3-common_3.24.5-1
libharfbuzz0b_2.3.1-1
libheif1_1.3.2-2~deb10u1
libhogweed4_3.4.1-1
libhtml-parser-perl_3.72-3+b3
libhtml-tagset-perl_3.20-3
libhtml-tree-perl_5.07-2
libhttp-cookies-perl_6.04-1
libhttp-date-perl_6.02-1
libhttp-message-perl_6.18-1
libhttp-negotiate-perl_6.01-1
libicu63_63.1-6
libidn11_1.33-2.2
libidn2-0_2.0.5-1
libio-html-perl_1.001-1
libio-socket-ssl-perl_2.060-3
libip4tc0_1.8.2-4
libisl19_0.20-2
libitm1_8.3.0-6
libjbig0_2.1-3.1+b2
libjpeg62-turbo_1:1.5.2-2+b1
libjson-c3_0.12.1+ds-2
libjson-glib-1.0-0_1.4.4-2
libjson-glib-1.0-common_1.4.4-2
libk5crypto3_1.17-3
libkeyutils1_1.6-6
libkmod2_26-1
libkrb5-3_1.17-3
libkrb5support0_1.17-3
liblcms2-2_2.9-3
liblqr-1-0_0.4.2-2.1
liblsan0_8.3.0-6
libltdl7_2.4.6-9
liblwp-mediatypes-perl_6.02-1
liblwp-protocol-https-perl_6.07-2
liblz4-1_1.8.3-1
liblzma5_5.2.4-1
libmagic-mgc_1:5.35-4
libmagic1_1:5.35-4
libmagickcore-6.q16-6_8:6.9.10.23+dfsg-2.1
libmagickwand-6.q16-6_8:6.9.10.23+dfsg-2.1
libmount1_2.33.1-0.1
libmpc3_1.1.0-1
libmpfr6_4.0.2-1
libmpx2_8.3.0-6
libncurses6_6.1+20181013-2
libncursesw6_6.1+20181013-2
libnet-http-perl_6.18-1
libnet-ssleay-perl_1.85-2+b1
libnettle6_3.4.1-1
libnuma1_2.0.12-1
libopenjp2-7_2.3.0-2
libp11-kit0_0.23.15-2
libpam-modules_1.3.1-5
libpam-modules-bin_1.3.1-5
libpam-runtime_1.3.1-5
libpam-systemd_241-5
libpam0g_1.3.1-5
libpango-1.0-0_1.42.4-6
libpangocairo-1.0-0_1.42.4-6
libpangoft2-1.0-0_1.42.4-6
libpcre3_2:8.39-12
libperl5.28_5.28.1-6
libpipeline1_1.5.1-2
libpixman-1-0_0.36.0-1
libpng16-16_1.6.36-6
libproxy1v5_0.4.15-5
libpsl5_0.20.2-2
libquadmath0_8.3.0-6
libreadline7_7.0-5
librest-0.7-0_0.8.1-1
librsvg2-2_2.44.10-2.1
librsvg2-bin_2.44.10-2.1
librsvg2-common_2.44.10-2.1
libseccomp2_2.3.3-4
libselinux1_2.8-1+b1
libsemanage-common_2.8-2
libsemanage1_2.8-2
libsepol1_2.8-1
libsigsegv2_2.12-2
libsmartcols1_2.33.1-0.1
libsoup-gnome2.4-1_2.64.2-2
libsoup2.4-1_2.64.2-2
libsqlite3-0_3.27.2-3
libss2_1.44.5-1
libssl1.1_1.1.1c-1
libstdc++-8-dev_8.3.0-6
libstdc++6_8.3.0-6
libsystemd0_241-5
libtasn1-6_4.13-3
libthai-data_0.1.28-2
libthai0_0.1.28-2
libtiff5_4.0.10-4
libtimedate-perl_2.3000-2
libtinfo6_6.1+20181013-2
libtool_2.4.6-9
libtry-tiny-perl_0.30-1
libtsan0_8.3.0-6
libubsan1_8.3.0-6
libuchardet0_0.0.6-3
libudev1_241-5
libunistring2_0.9.10-1
liburi-perl_1.76-1
libuuid1_2.33.1-0.1
libwayland-client0_1.16.0-1
libwayland-cursor0_1.16.0-1
libwayland-egl1_1.16.0-1
libwebp6_0.6.1-2
libwebpmux3_0.6.1-2
libwww-perl_6.36-2
libwww-robotrules-perl_6.02-1
libx11-6_2:1.6.7-1
libx11-data_2:1.6.7-1
libx265-165_2.9-4
libxau6_1:1.0.8-1+b2
libxcb-render0_1.13.1-2
libxcb-shm0_1.13.1-2
libxcb1_1.13.1-2
libxcomposite1_1:0.4.4-2
libxcursor1_1:1.1.15-2
libxdamage1_1:1.1.4-3+b3
libxdmcp6_1:1.1.2-3
libxext6_2:1.3.3-1+b2
libxfixes3_1:5.0.3-1
libxi6_2:1.7.9-1
libxinerama1_2:1.1.4-2
libxkbcommon0_0.8.2-1
libxml2_2.9.4+dfsg1-7+b3
libxrandr2_2:1.5.1-1
libxrender1_1:0.9.10-1
libzstd1_1.3.8+dfsg-3
linux-libc-dev_4.19.37-5
loadlin_1.6f-6
locales-all_2.28-10
login_1:4.5-1.1
lsb-base_10.2019051400
m4_1.4.18-2
make_4.2.1-1.2
man-db_2.8.5-2
mawk_1.3.3-17+b3
mingw-w64_6.0.0-3
mingw-w64-common_6.0.0-3
mingw-w64-i686-dev_6.0.0-3
mingw-w64-x86-64-dev_6.0.0-3
mount_2.33.1-0.1
ncurses-base_6.1+20181013-2
ncurses-bin_6.1+20181013-2
netbase_5.6
nsis_3.04-1
nsis-common_3.04-1
nsis-pluginapi_3.04-1
openssl_1.1.1c-1
passwd_1:4.5-1.1
patch_2.7.6-3
perl_5.28.1-6
perl-base_5.28.1-6
perl-modules-5.28_5.28.1-6
perl-openssl-defaults_3
po-debconf_1.0.21
readline-common_7.0-5
sbuild-build-depends-main-dummy_0.invalid.0
sed_4.7-1
sensible-utils_0.0.12
shared-mime-info_1.10-1
sudo_1.8.27-1
systemd_241-5
systemd-sysv_241-5
sysvinit-utils_2.93-8
tar_1.30+dfsg-6
tzdata_2019a-1
ucf_3.0038+nmu1
util-linux_2.33.1-0.1
vim_2:8.1.0875-5
vim-common_2:8.1.0875-5
vim-runtime_2:8.1.0875-5
xkb-data_2.26-2
xxd_2:8.1.0875-5
xz-utils_5.2.4-1
zlib1g_1:1.2.11.dfsg-1

[Didier 'OdyX' Raboud]

Hi there Simon,

thanks for the detailed investigation.

Le samedi, 24 août 2019, 11.52:53 h CEST Simon McVittie a écrit :
> On Fri, 23 Aug 2019 at 18:35:59 +0200, Didier 'OdyX' Raboud wrote:
> > I have uploaded win32-loader to buster to fix the out-of-sync archive
> > keys, but it has now repeatedly failed to build from source on the buster
> > buildds:
> >
> > https://buildd.debian.org/status/logs.php?pkg=win32-loader&ver=0.9.4%2Bdeb
> > 10u1
> >
> > I can't reproduce this in a local buster schroot, so I'm slightly at loss.
>
> I can reproduce this by trying to rebuild 0.9.4 in sbuild (a buster
> schroot created with sbuild-createchroot, hosted on a buster VM, using
> vectis[1]) if that's any help? Presumably there is some difference
> between your buster schroot and the one sbuild would use. See below
> for a full package list.

Yep. The difference I finally found was that the buildds use LANG=C.UTF-8 and
LC_ALL=C.UTF-8 whereas mine was not enforcing these (and so I was building
with LC_ALL=POSIX).

By fixing this, I could reproducibly fail to build win32-loader :-)

> > Unable to convert processed string "نوشتن ممکن نیست: " to codepage 1256
>
> That string appears to be part of the Farsi (fa) translation of nsis,
> found in "Contrib/Language files/Farsi.nlf" in nsis_3.04-1. It is indeed

> not possible to convert it to Windows codepage 1256: (…)

>
> (I don't know whether converting this string to CP1256 is an appropriate
> thing to be doing.)
>
> If that's the problem, maybe it would be possible to work around this by
> adjusting or disabling the Farsi translation, or by replacing the use of
> CP1256 with UTF-16 or something?

Hrm. For a stable upload, this seems to be enough to let the build go through:

--- a/debian/rules
+++ b/debian/rules
@@ -24,6 +24,10 @@ BUILT_USING_LIST := $(shell set -e; \

NSIS_VERSION := $(shell dpkg-query -f='$${Version}' -W nsis )

+# A non-UTF-8 locale is needed for the NSIS build to convert some language
strings
+LC_ALL := POSIX
+export LC_ALL
+
%:
dh $@

Any idea of a more targeted fix?

@Adam: I assume I need to bump the version number and upload again, right?
(debdiff attached)

Cheers,
OdyX

[Adam D. Barratt]

On Sat, 2019-08-24 at 18:49 +0200, Didier 'OdyX' Raboud wrote:
> @Adam: I assume I need to bump the version number and upload again,
> right?

Yep.

(I assume the changelog etc still end up generated as UTF-8.)

Regards,

Adam

[Simon McVittie]

On Sat, 24 Aug 2019 at 18:49:20 +0200, Didier 'OdyX' Raboud wrote:
> The difference I finally found was that the buildds use LANG=C.UTF-8 and
> LC_ALL=C.UTF-8 whereas mine was not enforcing these (and so I was building
> with LC_ALL=POSIX).

This was a change in sbuild 0.78.0, so in practice a difference between
buildds hosted on <= stretch (which used LC_ALL=POSIX) and buildds hosted
on >= buster (which use LC_ALL=C.UTF-8).

Forcing the POSIX (or equivalently C) locale effectively means you are
reverting a small part of the behaviour of newer sbuild, to be more like
the old sbuild where the win32-loader currently in buster was built,
so it seems a good minimal change to address this.

> +# A non-UTF-8 locale is needed for the NSIS build to convert some language
> strings
> +LC_ALL := POSIX
> +export LC_ALL

This is the first time I've encountered a package where changing the locale
to C.UTF-8 *causes* FTBFS - normally the failure mode is that unit tests
assume a UTF-8 (or at least legacy 8-bit) locale, and fail in LC_ALL=POSIX
(or equivalently LC_ALL=C).

It would be interesting to know whether the Farsi locale *works* in the
resulting build; but if it doesn't, then that probably isn't a regression,
because version 0.9.4 in buster would probably be broken in the same way.

smcv

[Didier 'OdyX' Raboud]

Le dimanche, 25 août 2019, 12.20:16 h CEST Simon McVittie a écrit :
> On Sat, 24 Aug 2019 at 18:49:20 +0200, Didier 'OdyX' Raboud wrote:
> > The difference I finally found was that the buildds use LANG=C.UTF-8 and
> > LC_ALL=C.UTF-8 whereas mine was not enforcing these (and so I was building
> > with LC_ALL=POSIX).
>
> This was a change in sbuild 0.78.0, so in practice a difference between
> buildds hosted on <= stretch (which used LC_ALL=POSIX) and buildds hosted
> on >= buster (which use LC_ALL=C.UTF-8).

Ah, thanks for the background.

> > +# A non-UTF-8 locale is needed for the NSIS build to convert some
> > language
> > strings
> > +LC_ALL := POSIX
> > +export LC_ALL
>
> This is the first time I've encountered a package where changing the locale
> to C.UTF-8 *causes* FTBFS - normally the failure mode is that unit tests
> assume a UTF-8 (or at least legacy 8-bit) locale, and fail in LC_ALL=POSIX
> (or equivalently LC_ALL=C).

win32-loader is special™ :-)

> It would be interesting to know whether the Farsi locale *works* in the
> resulting build; but if it doesn't, then that probably isn't a regression,
> because version 0.9.4 in buster would probably be broken in the same way.

I have not tested win32-loader in non-Wine win32 environments recently, so
we'd need someone with a Farsi Windows to test this.

But I see that Thomas is doing a lot of work on the experimental branch, and
we should let this hit unstable soon!

Cheers,
OdyX

[Didier 'OdyX' Raboud]

Control: clone -1 -2
Control: reopen -2 src:nsis
Control: retitle -2 NSIS: Farsi translation contains impossible conversions (YEH to CP1256)
Control: tags -2 upstream

Le samedi, 24 août 2019, 11.52:53 h CEST Simon McVittie a écrit :
> > Unable to convert processed string "نوشتن ممکن نیست: " to codepage 1256
>
> That string appears to be part of the Farsi (fa) translation of nsis,
> found in "Contrib/Language files/Farsi.nlf" in nsis_3.04-1. It is indeed
> not possible to convert it to Windows codepage 1256:
>
> $ python3
>
> >>> "نوشتن ممکن نیست".encode('cp1256')
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python3.7/encodings/cp1256.py", line 12, in encode
> return codecs.charmap_encode(input,errors,encoding_table)
> UnicodeEncodeError: 'charmap' codec can't encode character '\u06cc' in
> position 12: character maps to <undefined> $ unicode U+06CC
> U+06CC ARABIC LETTER FARSI YEH
>
> (I don't know whether converting this string to CP1256 is an appropriate
> thing to be doing.)
>
> If that's the problem, maybe it would be possible to work around this by
> adjusting or disabling the Farsi translation, or by replacing the use of
> CP1256 with UTF-16 or something?

Indeed. According to this thread on unicode-ml in 2001 [1], this seems to
only be supported in "not-exactly-CP1256" old Windows environments.
There's a similar bug in Pidgin [2], which brings the following comment [3]:

;; As this file needs to be encoded in CP1256 and CP1256 doesn't support U+06CC
;; and U+0654 characters, I have removed all U+0654 characters and replaced U+06CC
;; with U+064A in the middle of the words and with U+0649 at the end of the words.
;; The Presian text will display correctly but the encoding is incorrect.

It seems they entirely disabled persian translation for their installer now [4].

So; it seems to me that this is something that should be addressed in
NSIS (upstream), so cloning and reassigning, so that we have a reference bug to
point to.

Cheers,
OdyX

[1] https://unicode.org/mail-arch/unicode-ml/y2001-m10/0197.html NSIS
[2] https://developer.pidgin.im/ticket/2573
[3] https://developer.pidgin.im/attachment/ticket/2573/persian.2.nsh
[4] https://bitbucket.org/pidgin/main/src/7c5b54ec931b03b9354e0d2fffcac40b13e4aaa5/pidgin/win32/nsis/create_nsis_translations.pl#lines-110