[vim/vim] Compile-time race-condition during generation of ja.sjis.po (Issue #12826)
Etienne
Steps to reproduce
I am compiling vim as part of yocto poky version kirkstone, which is downloading vim's source-code using git (currently. The compilation works most of the time, but on some specific build-machine, the compilation of vim always fails. This is a combination of a racy build-system and the usage of a cross-compiler as CC variable:
- poky kirkstone currently uses version 9.0.1527 (commit c28e7a2 )
- the issue is that when the timestamp of the file src/po/ja.sjis.po is newer than the timestamp of the file src/po/ja.po, the Makefile src/po/Makefile ( https://github.com/vim/vim/blob/master/src/po/Makefile#L105 ) attempts to call "sjiscorr" in order to re-generate the file ja.sjis.po. However, the binary file sjiscorr is an ARM64 binary since I am cross-compiling Vim for ARM64. Thus, the compilation of vim fails.
Relying on the timestamp of src/po/ja.sjis.po is racy, because the timestamp of this file changes whenever git updates this file. So for instance when cloning the master branch and then checking out version 9.0.1527, the timestamp of ja.sjis.po is older than the timestamp of ja.po and Make will attempt to regenerate the file:
ecordonnier@lj8k2dq3:~/repos/vim$ ls -l src/po/ja.po src/po/ja.sjis.po
-rw-rw-r-- 1 ecordonnier ecordonnier 311785 Aug 17 17:28 src/po/ja.po
-rw-rw-r-- 1 ecordonnier ecordonnier 273989 Aug 17 17:28 src/po/ja.sjis.po
ecordonnier@lj8k2dq3:~/repos/vim$ git checkout v9.0.1527
...
ecordonnier@lj8k2dq3:~/repos/vim$ ls -l src/po/ja.po src/po/ja.sjis.po
-rw-rw-r-- 1 ecordonnier ecordonnier 311799 Aug 17 17:29 src/po/ja.po
-rw-rw-r-- 1 ecordonnier ecordonnier 273989 Aug 17 17:28 src/po/ja.sjis.po
Expected behaviour
- The build-system should not be racy
- Make should maybe not assume that sjiscorr compiled with CC can run on the host?
Version of Vim
9.0.1527
Environment
Ubuntu 22.04.
Poky version kirkstone
Logs and stack traces
| Starting make in the src directory. | If there are problems, cd to the src directory and run make there | cd src && make first | make[1]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src' | make[2]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src/po' | OLD_PO_FILE_INPUT=yes msgfmt -v -o cs.mo cs.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o fr.mo fr.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o ja.mo ja.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.mo pl.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o sk.mo sk.po | rm -f cs.cp1250.po | iconv -f utf-8 -t euc-jp ja.po | \ | LANG=C sed -e 's/charset=[uU][tT][fF]-8/charset=euc-jp/' -e 's/# Original translations/# Generated from ja.po, DO NOT EDIT/' > ja.euc-jp.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o ko.mo ko.po | iconv -f iso-8859-2 -t cp1250 cs.po | \ | LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from cs.po, DO NOT EDIT/' > cs.cp1250.po | rm -f pl.UTF-8.po | rm -f pl.cp1250.po | rm -f sk.cp1250.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_CN.mo zh_CN.po | iconv -f iso-8859-2 -t utf-8 pl.po | \ | LANG=C sed -e 's/charset=ISO-8859-2/charset=UTF-8/' -e 's/# Original translations/# Generated from pl.po, DO NOT EDIT/' > pl.UTF-8.po | iconv -f iso-8859-2 -t cp1250 pl.po | \ | LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from pl.po, DO NOT EDIT/' > pl.cp1250.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o zh_TW.mo zh_TW.po | iconv -f iso-8859-2 -t cp1250 sk.po | \ | LANG=C sed -e 's/charset=ISO-8859-2/charset=cp1250/' -e 's/# Original translations/# Generated from sk.po, DO NOT EDIT/' > sk.cp1250.po | OLD_PO_FILE_INPUT=yes msgfmt -v -o cs.cp1250.mo cs.cp1250.po | make[3]: Entering directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src/po' | ccache aarch64-snap-linux-gcc -march=armv8-a+crc -fstack-protector-strong -D_FORTIFY_SOURCE=1 -Wformat -Wformat-security -Werror=format-security --sysroot=/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/recipe-sysroot -o sjiscorr sjiscorr.c | 0 translated messages. | OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.cp1250.mo pl.cp1250.po | 0 translated messages. | OLD_PO_FILE_INPUT=yes msgfmt -v -o pl.UTF-8.mo pl.UTF-8.po | 0 translated messages. | 0 translated messages. | OLD_PO_FILE_INPUT=yes msgfmt -v -o sk.cp1250.mo sk.cp1250.po | 0 translated messages. | 0 translated messages. | 0 translated messages. | 0 translated messages. | 0 translated messages. | 0 translated messages. | 0 translated messages. | 2782 translated messages. | OLD_PO_FILE_INPUT=yes msgfmt -v -o ja.euc-jp.mo ja.euc-jp.po | In file included from /workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/bits/libc-header-start.h:33, | from /workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/stdio.h:27, | from sjiscorr.c:6: | /workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/recipe-sysroot/usr/include/features.h:412:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp] | 412 | # warning _FORTIFY_SOURCE requires compiling with optimization (-O) | | ^~~~~~~ | 2782 translated messages. | make[3]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src/po' | rm -f ja.sjis.po | iconv -f utf-8 -t cp932 ja.po | ./sjiscorr > ja.sjis.po | /bin/sh: ./sjiscorr: cannot execute binary file: Exec format error | make[2]: *** [Makefile:105: ja.sjis.po] Error 126 | make[2]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src/po' | make[1]: *** [Makefile:2074: languages] Error 2 | make[1]: Leaving directory '/workdir/build/tmp-glibc/work/armv8a-snap-linux/vim/9.0.1527-r0/git/src' | make: *** [Makefile:29: first] Error 2
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
K.Takata
- The build-system should not be racy
The commit e978b45 should have updated src/po/ja.sjis.po, however, it wasn't updated.
- Make should maybe not assume that sjiscorr compiled with CC can run on the host?
We should use HOSTCC or a similar variable, I think.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
I think I tried editing the file but my vim seems to have encoding issues. So I left it alone.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Etienne
- Make should maybe not assume that sjiscorr compiled with CC can run on the host?
We should use HOSTCC or a similar variable, I think.
FYI yocto provides "BUILD_CC" (for the host) and "CC" (for the target) in the environment. Yocto does not provide HOST_CC.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
K.Takata
I think I tried editing the file but my vim seemed to cause encoding issues. So I left it alone.
It will be automatically updated if you run make in src/po/.
I've done it in #12840.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
ah thanks!
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
Closed #12826 as completed via 0451056.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
K.Takata
- Make should maybe not assume that sjiscorr compiled with CC can run on the host?
This has not been fixed yet. Reopening.
@Ecordonnier
How did you run configure when you cross-compiling?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
K.Takata
Reopened #12826.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Etienne
@k-takata this is the code calling configure during a yocto build: https://github.com/yoctoproject/poky/blob/master/meta/recipes-support/vim/vim.inc#L41
do_configure () {
cd src
rm -f auto/*
touch auto/config.mk
aclocal
autoconf
cd ..
oe_runconf
touch src/auto/configure
touch src/auto/config.mk src/auto/config.h
}
I'm attaching a full log in case you want to have a look, but honestly if you are not familiar with yocto the complexity is high, so it may be easier to focus on a simpler minimal cross-compile use-case:
NOTE: Running ./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux --prefix=/usr --exec_prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/libexec --datadir=/usr/share --sysconfdir=/etc --sharedstatedir=/com --localstatedir=/var --libdir=/usr/lib --includedir=/usr/include --oldincludedir=/usr/include --infodir=/usr/share/info --mandir=/usr/share/man --disable-silent-rules --disable-dependency-tracking --with-libtool-sysroot=/home/ecordonnier/repos/poky/build/tmp/work/cortexa57-poky-linux/vim/9.0.1592/recipe-sysroot --disable-gpm --disable-gtktest --disable-xim --disable-netbeans --disable-desktop-database-update --with-tlib=ncurses --with-modified-by='Poky Maintainers po...@lists.yoctoproject.org' ac_cv_small_wchar_t=no ac_cv_path_GLIB_COMPILE_RESOURCES=no vim_cv_getcwd_broken=no vim_cv_memmove_handles_overlap=yes vim_cv_stat_ignores_slash=no vim_cv_terminfo=yes vim_cv_tgetent=non-zero vim_cv_toupper_broken=no vim_cv_tty_group=world STRIP=/bin/true --disable-static --enable-acl --enable-gui=gtk3 --enable-nls --disable-selinux --disable-canberra --with-features=big --with-x vim_cv_timer_create=yes
Full log:
log.do_configure.txt
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.