python3 and multibyte strings

[Stanislav Kondratyev]

Hello!

I’m reposting this issue from vim_multibyte thanks to John Beckett's
kind advice.

I was experimenting with the python3 interface and I stumbled upon the
following phenomenon. Suppose the cursor is in the string
абвгдеёжзийклмнопрстуфхцчшщъыьэюя
in a utf-8 encoded file. I run the following commands:
:py3 import vim
:py3 vim.current.line = vim.current.line
Surprisingly last 16½ of 33 characters disappear.

The same effect can be produced e. g. with
:py3 l = vim.current.line
:py3 vim.current.line = l
However, the string l can be correctly printed and sent to vim via
vim.command and setline. Hence I conclude that the transition from vim
to python does not cause any loss of information. On the contrary,
when the string is being passed from python to vim, vim (or python?)
calculates that it is 33 ‘units’ long and copies 33 *bytes*.

Experiments with less appaulling accented Latin letters show the same
effect. For example,
when the command
:py3 vim.current.line = vim.current.line
is applied to the line ‘aàáâãä', it yields
aàá<c3>

I’m running vim 7.3 on Windows 7.

I’d like to find out whether this is a common effect and whether there
exists an elegant way of changing multibyte strings from python3.

Any help is appreciated. :)

Stanislav

[lilydjwg]

I can't reproduce it on Windows XP with cp936 and Linux with UTF-8 locale.
What's the value of your 'encoding' option and the exact version of your
Vim?

--
Best regards,
lilydjwg

Linux Vim Python 我的博客:
http://lilydjwg.is-programmer.com/
--
A: Because it obfuscates the reading.
Q: Why is top posting so bad?

[Stanislav Kondratyev]

On 6 май, 15:19, lilydjwg <lilyd...@gmail.com> wrote:
> I can't reproduce it on Windows XP with cp936 and Linux with UTF-8 locale.
> What's the value of your 'encoding' option and the exact version of your
> Vim?
>
> --
> Best regards,
> lilydjwg
>
> Linux Vim Python 我的博客:http://lilydjwg.is-programmer.com/
> --
> A: Because it obfuscates the reading.
> Q: Why is top posting so bad?

My Vim is

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 27 2010 17:59:02)
MS-Windows 32-bit GUI version with OLE support
Included patches: 1-46
Compiled by Bram@KIBAALE
Big version with GUI.

I’ve set the encoding to utf-8 in the vimrc.

Sorry, is cp936 a multibyte encoding?

[lilydjwg]

On Mon, May 07, 2012 at 01:36:44PM -0700, Stanislav Kondratyev wrote:
> My Vim is
>
> VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 27 2010 17:59:02)
> MS-Windows 32-bit GUI version with OLE support
> Included patches: 1-46
> Compiled by Bram@KIBAALE
> Big version with GUI.
>
> I’ve set the encoding to utf-8 in the vimrc.
>
> Sorry, is cp936 a multibyte encoding?

Yes.

I think you need to get a newer Vim. This was fixed around patch number 200.

[Tony Mechelynck]

On 08/05/12 07:13, lilydjwg wrote:
> On Mon, May 07, 2012 at 01:36:44PM -0700, Stanislav Kondratyev wrote:
>> My Vim is
>>
>> VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 27 2010 17:59:02)
>> MS-Windows 32-bit GUI version with OLE support
>> Included patches: 1-46
>> Compiled by Bram@KIBAALE
>> Big version with GUI.
>>

>> I嚙踝蕭ve set the encoding to utf-8 in the vimrc.

>>
>> Sorry, is cp936 a multibyte encoding?
>
> Yes.
>
> I think you need to get a newer Vim. This was fixed around patch number 200.
>

FYI, the latest source is 7.3.515.

Best regards,
Tony.
--
Cigarette, n.:
A fire at one end, a fool at the other, and a bit of tobacco in
between.

[Stanislav Kondratyev]

Many thanks for the tips! Now it works.

I failed to found a recent Windows installer with python 3 enabled, so I had to build gvim myself (and I’m especially proud of it. :)

In case someone googles the same issue, I’d like to summarize my experience:
http://vim.wikia.com/wiki/Build_Python-enabled_Vim_on_Windows_with_MinGW
I edited Make_ming.mak and Makefile in their part concerning python. One can build gvim with the following command:
mingw32-make -f Make_ming.mak gvim.exe