Non-ASCII characters problem

[Michael]

Hello,

Here is a string from the input file:

Кристалл(микросхема) № 1 Пластина № 0000 Партия № 7

The program reads the number of the element:

read(1,'(A)',err=31,end=31)str
if(index(trim(str),'Кристалл(микросхема)').ne.0) then
el_cnt=el_cnt+1
j=index(str,'№')
read(str(j+2:),*)numbers(el_cnt)
end if

System is Windows 7, 64b. The program compiled with g95/mingw executes
without errors, but the program compiled with gfortran gives an error:
End of file.

If I change str(j+2:) to str(j+2:j+5) then it gives no errors. It seems
that program stumbles upon the Cyrillic character "П".

Is this a bug, or there is something wrong in the program?


(P.S. See this in UTF-8)

[michael...@compuserve.com]

You don't show us the declaration of str. Is it of kind=ucs4 (MFE, pp. 310-311)?

Regards,

Mike Metcalf

[Tobias Burnus]

Michael wrote:
> Here is a string from the input file:
>
> Кристалл(микросхема) № 1 Пластина № 0000 Партия № 7

In UTF-8 encoding, every Cyrillic letter and "№" needs more than a byte,
the 0, 1, 7, " ", "(" and ")" need one byte.

> The program reads the number of the element:
> read(1,'(A)',err=31,end=31)str
> if(index(trim(str),'Кристалл(микросхема)').ne.0) then
> el_cnt=el_cnt+1
> j=index(str,'№')
> read(str(j+2:),*)numbers(el_cnt)

Trying your code shows, it works if you use "j+3" instead of "j+2". You
can see it yourself if you write:

write(*,*) str(j+2:)
or
write(*,*) str(j+3:)


In principle, it would be better to use Fortran 2003's Unicode support.
In that case, the compiler already knows that the input is in UTF-8 and
one character in the string matches a single character - without the
need to worry about the the number of bytes.

gfortran (since GCC 4.4) handles Unicode; unfortunately, it currently
does not allow one to enter Unicode characters directly in the input.
The following program works, but I admit it is a bit ugly with the
backslashes - and the backslash notation is admittedly also not fully
standard conform (but more readable that char(...,kind=...)).

The support of directly inputting UTF-8 characters is on the to-do list,
but unfortunately, that list is rather long. (Additional contributors
are highly welcome; thus, if you (or someone else) is interested …)

Tobias


! compile with "gfortran -fbackslash"

program test
use iso_fortran_env, only: output_unit
implicit none
integer, parameter :: ucs4 = selected_char_kind ('ISO_10646')
character(kind=ucs4), parameter :: &
No_char = char(int(z'2116'),kind=ucs4) ! = №
character(kind=ucs4, len=100) :: str
integer :: j, numbers

open (output_unit, encoding='UTF-8')

open (99, file="test.dat", encoding='UTF-8')
read(99,'(A)')str

if(index(trim(str), & ! 'Кристалл(микросхема)'
ucs4_'\u041A\u0440\u0438\u0441\u0442\u0430\u043B\u043B(' &
//ucs4_'\u043C\u0438\u043A\u0440\u043E\u0441\u0445\u0435\u043C\u0430)'&
).ne.0) then
j=index(str,No_char)
! print *, 'Debug output:',trim(str(j+2:))
read(str(j+2:),*)numbers
end if

print *, numbers
end

[michael....@gmail.com]

среда, 15 мая 2013 г., 19:48:34 UTC+7 пользователь Tobias Burnus написал:

No, my encoding is not UTF-8, but 1-byte CP1251. It's just ny mail client is set to UTF-8.
BTW, I tried to change the rest of the line to a random set of Cyrillic letters, and the program works. Only the combination from my first post gives an error.

[glen herrmannsfeldt]

michael....@gmail.com wrote:
> ??????????, 15 ?????? 2013ￜ??., 19:48:34 UTC+7 ???????????????????????? Tobias Burnus ??????????????:

>> Michael wrote:

>> > Here is a string from the input file:

>> > ????????????????(????????????????????) ??? 1 ???????????????? ??? 0000 ???????????? ??? 7

>> In UTF-8 encoding, every Cyrillic letter and "???" needs more than a byte,

>> the 0, 1, 7, " ", "(" and ")" need one byte.

>> > The program reads the number of the element:

>> > read(1,'(A)',err=31,end=31)str

>> > if(index(trim(str),'????????????????(????????????????????)').ne.0) then
>> > el_cnt=el_cnt+1
>> > j=index(str,'???')

>> > read(str(j+2:),*)numbers(el_cnt)

>> Trying your code shows, it works if you use "j+3" instead of "j+2". You
>> can see it yourself if you write:

>> write(*,*) str(j+2:)

>> or

>> write(*,*) str(j+3:)

(snip)

> No, my encoding is not UTF-8, but 1-byte CP1251.
> It's just ny mail client is set to UTF-8.
> BTW, I tried to change the rest of the line to a random set
> of Cyrillic letters, and the program works.
> Only the combination from my first post gives an error.

You could

print *,j,str(j+2,:)

to see what exactly is being passed to the read.

Also, printing j helps to debug in case the index isn't doing what
you think it is.

print *,char('???')

where that is the character in the last index call.
(I am not sure I can post UTF-8, so it might not go through.)

-- glen

[glen herrmannsfeldt]

michael....@gmail.com wrote:


You might try:

print *,char('№')

along with some of the other characters.

(And I now set UTF-8 in my newsreader, so it should go through.)

-- glen

[Michael]

> You could
>
> print *,j,str(j+2,:)
>
> to see what exactly is being passed to the read.
>
> Also, printing j helps to debug in case the index isn't doing what
> you think it is.
>
> print *,char('???')
>
> where that is the character in the last index call.
> (I am not sure I can post UTF-8, so it might not go through.)
>
> -- glen
>

The output contains one space and the number that SHOULD BE read to the
variable, then the rest of the string and then spaces till the length of
'str'. Still seems as a bug.

[William Clodius]

<michael....@gmail.com> wrote:

> <snip>

>
> No, my encoding is not UTF-8, but 1-byte CP1251. It's just ny mail client
> is set to UTF-8.
> BTW, I tried to change the rest of the line to a random set of Cyrillic
> letters, and the program works. Only the combination from my first
> post gives an error.

Compilers as a rule do not independently know the encoding in a text
file, they assume an encoding based on various settings. My suspicion is
that gfortran, given its wide usage in the Linux, by default assumes
that the encoding of a text file is UTF-8, but by setting the
appropriate locale you might be able to override that default. To
understand the interpretation of the file it might be useful to examine
it not as text, but as its internal representation, a sequence of eight
bit integers. something equivalent to:

integer(byte_kind) :: byte
open(lun, text_filename, access='stream', status='old')
do
read(lun, EOF=999) byte
write(*, fmt='(I4)') modulo(int(byte, integer_kind), 256)
end do
999 continue

[Михаил Окунцов]

> Compilers as a rule do not independently know the encoding in a text
> file, they assume an encoding based on various settings. My suspicion is
> that gfortran, given its wide usage in the Linux, by default assumes
> that the encoding of a text file is UTF-8, but by setting the
> appropriate locale you might be able to override that default.  To
> understand the interpretation of the file it might be useful to examine
> it not as text, but as its internal representation, a sequence of eight
> bit integers. something equivalent to:
>
> integer(byte_kind) :: byte
> open(lun, text_filename, access='stream', status='old')
> do
>       read(lun, EOF=999) byte
>       write(*, fmt='(I4)') modulo(int(byte, integer_kind), 256)
> end do
> 999 continue

No, the encoding is definitely CP1251.

[Tobias Burnus]

William Clodius wrote:
> Compilers as a rule do not independently know the encoding in a text
> file, they assume an encoding based on various settings. My suspicion is
> that gfortran, given its wide usage in the Linux, by default assumes
> that the encoding of a text file is UTF-8,

No, the compiler assumes by default that the file consists of characters
which are 8 bits wide. That's the case for both the source file and - in
particular - for Fortran I/O.

For files accessed via Fortran, Fortran 2003 offers a way to change the
encoding to UTF-8 ('open(...,encoding="UTF-8")').

For the source file, read by gfortran, it would be in principle possible
to honour the locale (e.g. UTF-8 etc.) but for some reason that
currently doesn't work. (The used libcpp library supports it and also
internally the compiler supports it, but something is missing to make it
really work.)

While I believe UTF-8 is the future (at least on Unix-based systems like
Linux or OS X - Microsoft seems to prefer UCS2), I don't think that this
is the problem here. Codepage 1251 uses a 8bit characters and shouldn't
be handled any different from the (7bit) ASCII or (8bit) Latin 1.


I have frankly no idea what's special about an 8bit "П". For "№ 1
П", list-directed read (if placed after the "No" (№) should only read up
to the " " (blank) after the "1". Additionally, I also do not see what's
special about that character - it shouldn't be mistaken for a number, or
any of the other special characters (blank, tab, "/", line break (CR,LF)
etc.).

Tobias