Patch 8.1.1468

[Bram Moolenaar]

Patch 8.1.1468
Problem: The generated desktop files may be invalid.
Solution: Check validity with desktop-file-validate. (Christian Brabandt,
Will Thompson, closes #4480)
Files: src/po/Makefile


*** ../vim-8.1.1467/src/po/Makefile 2019-05-05 21:01:47.654072636 +0200
--- src/po/Makefile 2019-06-04 23:10:49.853378460 +0200
***************
*** 151,157 ****

clean: checkclean
rm -f core core.* *.old.po *.mo *.pot sjiscorr
! rm -f LINGUAS vim.desktop gvim.desktop

distclean: clean

--- 151,157 ----

clean: checkclean
rm -f core core.* *.old.po *.mo *.pot sjiscorr
! rm -f LINGUAS vim.desktop gvim.desktop tmp_*desktop

distclean: clean

***************
*** 167,179 ****

vim.desktop: vim.desktop.in $(POFILES)
@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
! $(MSGFMT) --desktop -d . --template vim.desktop.in -o vim.desktop
rm -f LINGUAS

gvim.desktop: gvim.desktop.in $(POFILES)
@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
! $(MSGFMT) --desktop -d . --template gvim.desktop.in -o gvim.desktop
rm -f LINGUAS

update-po: $(LANGUAGES)

--- 167,183 ----

vim.desktop: vim.desktop.in $(POFILES)
@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
! $(MSGFMT) --desktop -d . --template vim.desktop.in -o tmp_vim.desktop
rm -f LINGUAS
+ if command -v desktop-file-validate; then desktop-file-validate tmp_vim.desktop; fi
+ mv tmp_vim.desktop vim.desktop

gvim.desktop: gvim.desktop.in $(POFILES)
@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
! $(MSGFMT) --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
rm -f LINGUAS
+ if command -v desktop-file-validate; then desktop-file-validate tmp_gvim.desktop; fi
+ mv tmp_gvim.desktop gvim.desktop

update-po: $(LANGUAGES)

*** ../vim-8.1.1467/src/version.c 2019-06-04 23:20:19.974043597 +0200
--- src/version.c 2019-06-05 21:25:12.494834304 +0200
***************
*** 769,770 ****
--- 769,772 ----
{ /* Add new patch number below this line */
+ /**/
+ 1468,
/**/

--
I AM THANKFUL...
...for the mess to clean after a party because it means I have
been surrounded by friends.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Mike Williams]

Hi

On 05/06/2019 20:25, Bram Moolenaar wrote:
>
> Patch 8.1.1468
> Problem: The generated desktop files may be invalid.
> Solution: Check validity with desktop-file-validate. (Christian Brabandt,
> Will Thompson, closes #4480)
> Files: src/po/Makefile
>

A clean build and install is now reporting the following warnings and
errors:

msgfmt --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
rm -f LINGUAS

if command -v desktop-file-validate; then desktop-file-validate
tmp_gvim.desktop; fi

/usr/bin/desktop-file-validate
tmp_gvim.desktop: warning: file contains lines that are not UTF-8
encoded. There is no guarantee the validator will correctly work.
tmp_gvim.desktop: error: file contains multiple keys named
"GenericName[eo]" in group "Desktop Entry"
tmp_gvim.desktop: error: file contains multiple keys named "Comment[eo]"
in group "Desktop Entry"
[Invalid UTF-8] tmp_gvim.desktop: error: value
"\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa0\xe0\xad\x8d\xe0\xac\xaf
\xe0\xac\xab\xe0\xac\xbe\xe0\xac\x87\xe0\xac\xb2\xe0\xac\x97\xe0\xad\x81\xe0\xac\xa1\xe0\xac\xbc\xe0\xac\xbf\xe0\xac\x95\xe0\xad\x81
\xe0\xac\xb8\xe0\xac\xae\xe0\xad\x8d\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa6\xe0\xac\xa8
\xe0\xac\x95\xe0\xac\xb0\xe0\xac\xa8\xe0\xad\x8d\xe0\xac\xa4\xe0\xad"
for locale string key "Comment[or]" in group "Desktop Entry" contains
invalid UTF-8 characters, locale string values should be encoded in UTF-8
Makefile:176: recipe for target 'gvim.desktop' failed
make[2]: *** [gvim.desktop] Error 1
make[2]: Leaving directory '/home/mikew/repos/vim/src/po'
Makefile:2021: recipe for target 'languages' failed
make[1]: *** [languages] Error 2
make[1]: Leaving directory '/home/mikew/repos/vim/src'
Makefile:26: recipe for target 'first' failed
make: *** [first] Error 2

This is on Ubuntu 16.04 Is this a problem with the VIM po source or
lack or installed tooling?

TTFN

Mike
--
Why do they call it the rush hour when nothing moves?

[Christian Brabandt]

On Do, 06 Jun 2019, Mike Williams wrote:

> Hi
>
> On 05/06/2019 20:25, Bram Moolenaar wrote:
> >
> > Patch 8.1.1468
> > Problem: The generated desktop files may be invalid.
> > Solution: Check validity with desktop-file-validate. (Christian Brabandt,
> > Will Thompson, closes #4480)
> > Files: src/po/Makefile
> >
>
> A clean build and install is now reporting the following warnings and
> errors:
>
> msgfmt --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
> rm -f LINGUAS
> if command -v desktop-file-validate; then desktop-file-validate
> tmp_gvim.desktop; fi
> /usr/bin/desktop-file-validate

> tmp_gvim.desktop: warning: file contains lines that are not UTF-8
> encoded. There is no guarantee the validator will correctly work.

I have no clue where that comes from.

> tmp_gvim.desktop: error: file contains multiple keys named
> "GenericName[eo]" in group "Desktop Entry"
> tmp_gvim.desktop: error: file contains multiple keys named "Comment[eo]"
> in group "Desktop Entry"

That is expected and fixed by
https://github.com/vim/vim/commit/61da1bfa6c6b19dd670671a318ce9f9e2acc784c

> [Invalid UTF-8] tmp_gvim.desktop: error: value
> "\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa0\xe0\xad\x8d\xe0\xac\xaf
> \xe0\xac\xab\xe0\xac\xbe\xe0\xac\x87\xe0\xac\xb2\xe0\xac\x97\xe0\xad\x81\xe0\xac\xa1\xe0\xac\xbc\xe0\xac\xbf\xe0\xac\x95\xe0\xad\x81
> \xe0\xac\xb8\xe0\xac\xae\xe0\xad\x8d\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa6\xe0\xac\xa8
> \xe0\xac\x95\xe0\xac\xb0\xe0\xac\xa8\xe0\xad\x8d\xe0\xac\xa4\xe0\xad"
> for locale string key "Comment[or]" in group "Desktop Entry" contains
> invalid UTF-8 characters, locale string values should be encoded in UTF-8

I don't see that. I suspect either msgfmt cannot correctly handle the
template (gvim.desktop.in) file or desktop-file-validate does
incorrectly detect it as non UTF-8

BTW, is your locale set correctly?

For the records, in my debian dev box, I am using:
gettext:
Installed: 0.19.8.1-9
Candidate: 0.19.8.1-9
Version table:
*** 0.19.8.1-9 500
500 http://ftp.de.debian.org/debian buster/main amd64 Packages
100 /var/lib/dpkg/status
0 140901 chrisbra@debian ~/code/vim-src/src/po (git)-[desktop-file-utils]- % apt-cache policy desktop-file-utils
desktop-file-utils:
Installed: 0.23-4
Candidate: 0.23-4
Version table:
*** 0.23-4 500
500 http://ftp.de.debian.org/debian buster/main amd64 Packages
100 /var/lib/dpkg/status

> This is on Ubuntu 16.04 Is this a problem with the VIM po source or
> lack or installed tooling?

I am not sure yet. What does the raw generated utf-8 file tmp_gvim.desktop look like?

Mit freundlichen Grüßen
Christian
--
Für das Wohlbefinden einer Frau sind bewundernde Männerblicke
wichtiger als Kalorien und Medikamente.
-- Françoise Sagan

[Mike Williams]

On 06/06/2019 12:16, Christian Brabandt wrote:
>
> On Do, 06 Jun 2019, Mike Williams wrote:
>
>> Hi
>>
>> On 05/06/2019 20:25, Bram Moolenaar wrote:
>>>
>>> Patch 8.1.1468
>>> Problem: The generated desktop files may be invalid.
>>> Solution: Check validity with desktop-file-validate. (Christian Brabandt,
>>> Will Thompson, closes #4480)
>>> Files: src/po/Makefile
>>>
>>
>> A clean build and install is now reporting the following warnings and
>> errors:
>>
>> msgfmt --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
>> rm -f LINGUAS
>> if command -v desktop-file-validate; then desktop-file-validate
>> tmp_gvim.desktop; fi
>> /usr/bin/desktop-file-validate
>
>> tmp_gvim.desktop: warning: file contains lines that are not UTF-8
>> encoded. There is no guarantee the validator will correctly work.
>
> I have no clue where that comes from.

This may be related to the later UTF-8 warning.

>> tmp_gvim.desktop: error: file contains multiple keys named
>> "GenericName[eo]" in group "Desktop Entry"
>> tmp_gvim.desktop: error: file contains multiple keys named "Comment[eo]"
>> in group "Desktop Entry"
>
> That is expected and fixed by
> https://github.com/vim/vim/commit/61da1bfa6c6b19dd670671a318ce9f9e2acc784c

Confirmed.

>> [Invalid UTF-8] tmp_gvim.desktop: error: value
>> "\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa0\xe0\xad\x8d\xe0\xac\xaf
>> \xe0\xac\xab\xe0\xac\xbe\xe0\xac\x87\xe0\xac\xb2\xe0\xac\x97\xe0\xad\x81\xe0\xac\xa1\xe0\xac\xbc\xe0\xac\xbf\xe0\xac\x95\xe0\xad\x81
>> \xe0\xac\xb8\xe0\xac\xae\xe0\xad\x8d\xe0\xac\xaa\xe0\xac\xbe\xe0\xac\xa6\xe0\xac\xa8
>> \xe0\xac\x95\xe0\xac\xb0\xe0\xac\xa8\xe0\xad\x8d\xe0\xac\xa4\xe0\xad"
>> for locale string key "Comment[or]" in group "Desktop Entry" contains
>> invalid UTF-8 characters, locale string values should be encoded in UTF-8
>
> I don't see that. I suspect either msgfmt cannot correctly handle the
> template (gvim.desktop.in) file or desktop-file-validate does
> incorrectly detect it as non UTF-8
>
> BTW, is your locale set correctly?

I believe so. All settings are en_GB.UTF-8

> For the records, in my debian dev box, I am using:
> gettext:
> Installed: 0.19.8.1-9
> Candidate: 0.19.8.1-9
> Version table:
> *** 0.19.8.1-9 500
> 500 http://ftp.de.debian.org/debian buster/main amd64 Packages
> 100 /var/lib/dpkg/status
> 0 140901 chrisbra@debian ~/code/vim-src/src/po (git)-[desktop-file-utils]- % apt-cache policy desktop-file-utils
> desktop-file-utils:
> Installed: 0.23-4
> Candidate: 0.23-4
> Version table:
> *** 0.23-4 500
> 500 http://ftp.de.debian.org/debian buster/main amd64 Packages
> 100 /var/lib/dpkg/status

~/repos/vim $ apt list gettext desktop-file-utils
Listing... Done
desktop-file-utils/xenial-updates,now 0.22-1ubuntu5.2 i386 [installed]
gettext/xenial-updates,xenial-security,now 0.19.7-2ubuntu3.1 i386
[installed]

So slightly behind your versions.

>> This is on Ubuntu 16.04 Is this a problem with the VIM po source or
>> lack or installed tooling?
>
> I am not sure yet. What does the raw generated utf-8 file tmp_gvim.desktop look like?

It looks like the last byte of the last character has been dropped in
processing. The .in file has the UTF-8 sequence 0xe0 0xad 0x81 0x0a for
the last character and the newline, but the tmp file has 0xe0 0xad 0x0a
which is an invalid UTF-8 sequence. Removing the line from both the .in
files allows make to run to completion with no errors.

This looks like a bug in msgfmt which should not be emitting invalid
UTF-8 sequences. In the meantime this will break all builds on U16.04. :(

TTFN

Mike
--
If you're not making waves, you're not rowing the boat.

[Christian Brabandt]

Indeed. Try reporting a bug to ubuntu, though I doubt they will upload a
new (fixed) version. In the meantime, you might want to use this package
here:
https://launchpad.net/~ricotz/+archive/ubuntu/toolchain

We are using this on our Travis build script as well.

Best,
Christian
--
Wie man sein Kind nicht nennen sollte:
Bud Ismus

[Mike Williams]

16.04 is a Ubuntu LTS and supported for another 18 months or so. It
would be nice to think they would want to updated for such an obvious
bug on release widely used, but who knows.

This is for a work machine so modifying toolchains via a PPA is not
possible. I can work around the problem now that I know what it is.

TTFN

Mike
--
The wind of change blows straight into the face of time.

[Christian Brabandt]

On Do, 06 Jun 2019, Mike Williams wrote:

> 16.04 is a Ubuntu LTS and supported for another 18 months or so. It
> would be nice to think they would want to updated for such an obvious
> bug on release widely used, but who knows.

Just for the fun of it, I have been trying to see what commit caused it.
I think, this commit fixed it:
http://git.savannah.gnu.org/cgit/gettext.git/commit/?id=68ab0dafa99f1941b3ebb47b7cf969381e7310f4

Going further I could successfully apply that patch against Ubuntus
16.04 LTS version and rebuild the complete package. That seems to have
fixed the problem on my local xenial (16.04LTS) box. So perhaps you can
convince them to rebuild with that patch applied on top.

> This is for a work machine so modifying toolchains via a PPA is not
> possible. I can work around the problem now that I know what it is.

Understood. I have been wondering if it would be doable to detect this
problem somewhere and then skip this step. Not sure how to do this and
if it would be worth the trouble.

Best,
Christian
--
Sobald der Glaube an die Allmacht physischer Gewalt das Leben einer Nation
beherrscht, emanzipiert sich diese Kraft, wird zum Selbstzweck und wird
mächtiger als jene Männer, die sie als Werkzeug benützen wollen.
-- Albert Einstein

[Mike Williams]

On 06/06/2019 16:48, Christian Brabandt wrote:
>
> On Do, 06 Jun 2019, Mike Williams wrote:
>
>> 16.04 is a Ubuntu LTS and supported for another 18 months or so. It
>> would be nice to think they would want to updated for such an obvious
>> bug on release widely used, but who knows.
>
> Just for the fun of it, I have been trying to see what commit caused it.
> I think, this commit fixed it:
> http://git.savannah.gnu.org/cgit/gettext.git/commit/?id=68ab0dafa99f1941b3ebb47b7cf969381e7310f4
>
> Going further I could successfully apply that patch against Ubuntus
> 16.04 LTS version and rebuild the complete package. That seems to have
> fixed the problem on my local xenial (16.04LTS) box. So perhaps you can
> convince them to rebuild with that patch applied on top.

On a straight read of the patch that seems a surprising fix, but I'll
happily take your word for it.

>> This is for a work machine so modifying toolchains via a PPA is not
>> possible. I can work around the problem now that I know what it is.
>
> Understood. I have been wondering if it would be doable to detect this
> problem somewhere and then skip this step. Not sure how to do this and
> if it would be worth the trouble.

The desktop file still seems valid apart from encoding errors. Is there
a problem if the makefile ignored the error and continued? Or have a
build option to allow ignoring errors generating desktop files? Just a
wild thought.

TTFN

Mike
--
Everyone I know lives on a fold on a map!

[Christian Brabandt]

On Do, 06 Jun 2019, Mike Williams wrote:

> The desktop file still seems valid apart from encoding errors. Is there
> a problem if the makefile ignored the error and continued? Or have a

The problem is, we might not notice failures :(

> build option to allow ignoring errors generating desktop files? Just a
> wild thought.

I think for fixing your local environment, the best is to commit a local
patch like below, that should just ignore the error. Granted, that would
install a probably invalid desktop file, but that happened before as
well (and it seemed nobody noticed since the desktop files are generated
from the po files directly (8.1.1147)).

Well, at least Debian and Ubuntu do not offer official packages newer
than 8.1.0875, in case of Debian this is probably due to the buster
freeze).

That patch could be stashed on top, or just kept around in your local
repository, shouldn't really matter.

diff --git a/src/po/Makefile b/src/po/Makefile
index 8c7066b64..dcf88ab19 100644
--- a/src/po/Makefile
+++ b/src/po/Makefile
@@ -169,14 +169,14 @@ vim.desktop: vim.desktop.in $(POFILES)

@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS

$(MSGFMT) --desktop -d . --template vim.desktop.in -o tmp_vim.desktop
rm -f LINGUAS

- if command -v desktop-file-validate; then desktop-file-validate tmp_vim.desktop; fi
+ -if command -v desktop-file-validate; then desktop-file-validate tmp_vim.desktop; fi

mv tmp_vim.desktop vim.desktop

gvim.desktop: gvim.desktop.in $(POFILES)
@echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS

$(MSGFMT) --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
rm -f LINGUAS
- if command -v desktop-file-validate; then desktop-file-validate tmp_gvim.desktop; fi
+ -if command -v desktop-file-validate; then desktop-file-validate tmp_gvim.desktop; fi

mv tmp_gvim.desktop gvim.desktop

update-po: $(LANGUAGES)

Best,
Christian
--
Eine Frau heiratet einen Mann in der Erwartung, daß er sich ändert,
aber er ändert sich nicht.
Ein Mann heiratet eine Frau in der Erwartung, daß sie sich nicht
ändert - doch sie tut es.

[Mike Williams]

On 06/06/2019 18:22, Christian Brabandt wrote:
>
> On Do, 06 Jun 2019, Mike Williams wrote:
>
>> The desktop file still seems valid apart from encoding errors. Is there
>> a problem if the makefile ignored the error and continued? Or have a
>
> The problem is, we might not notice failures :(

You still get the error reported, but yes I see your point. Fine for
local development, not for CI where you want it to pick up errors.

>> build option to allow ignoring errors generating desktop files? Just a
>> wild thought.
>
> I think for fixing your local environment, the best is to commit a local
> patch like below, that should just ignore the error. Granted, that would
> install a probably invalid desktop file, but that happened before as
> well (and it seemed nobody noticed since the desktop files are generated
> from the po files directly (8.1.1147)).
>
> Well, at least Debian and Ubuntu do not offer official packages newer
> than 8.1.0875, in case of Debian this is probably due to the buster
> freeze).

U16.04 is 7.4.1689. The level of pain may depend on how many people
build recent versions of VIM or provide a PPA for others.

> That patch could be stashed on top, or just kept around in your local
> repository, shouldn't really matter.
>
> diff --git a/src/po/Makefile b/src/po/Makefile
> index 8c7066b64..dcf88ab19 100644
> --- a/src/po/Makefile
> +++ b/src/po/Makefile
> @@ -169,14 +169,14 @@ vim.desktop: vim.desktop.in $(POFILES)
> @echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
> $(MSGFMT) --desktop -d . --template vim.desktop.in -o tmp_vim.desktop
> rm -f LINGUAS
> - if command -v desktop-file-validate; then desktop-file-validate tmp_vim.desktop; fi
> + -if command -v desktop-file-validate; then desktop-file-validate tmp_vim.desktop; fi
> mv tmp_vim.desktop vim.desktop
>
> gvim.desktop: gvim.desktop.in $(POFILES)
> @echo $(LANGUAGES) | tr " " "\n" |sed -e '/\./d' | sort > LINGUAS
> $(MSGFMT) --desktop -d . --template gvim.desktop.in -o tmp_gvim.desktop
> rm -f LINGUAS
> - if command -v desktop-file-validate; then desktop-file-validate tmp_gvim.desktop; fi
> + -if command -v desktop-file-validate; then desktop-file-validate tmp_gvim.desktop; fi
> mv tmp_gvim.desktop gvim.desktop
>
> update-po: $(LANGUAGES)

I still run off an hg repo so a simple mq patch will quickly do what I
need. Thanks for all the help.

TTFN

Mike
--
If you throw a cat out of a car window... is it kitty litter??