Exclusive file access

[ah...@marriott.org]

Dear All,

Obviously I'm misunderstanding something here.

I thought that if two processes tried to open the same file for write access then the second process would get an exception.

I have a simple example which when compiled using GnatPro 7.3.1 and run under 32-bit Windows XP demonstrates that this is not so.

Which then begs the question on how to detect if the file is in use and abort if it is.
This should be simple but obviously too complex for my tiny brain.
Does anyone know how to do this from Ada (without resorting to the OS directly)?

My simple test program is

package body Test is

package Io renames Ada.Text_IO;

procedure Work is
The_File : Io.File_Type;
begin
Io.Open (The_File, Io.Out_File, "Test.Txt");
for Count in Natural'value(Ada.Command_Line.Argument(1)) .. Natural'value(Ada.Command_Line.Argument(2)) loop
Io.Put_Line (The_File, "Count =" & Count'img);
Io.Put_Line ("Count =" & Count'img);
delay 1.0;
end loop;
Io.Close (The_File);
exception
when others =>
Io.Put_Line ("Exception");
end Work;

end Test;

If I execute this from one process with parameters 1 10 and then when it reaches 8 start the program again from a second process with parameters 3 7 on the same machine, the resultant file is a nice mixture! :-(

Count = 3
Count = 4
Count = 5
Count = 6
Count = 7
Count = 6
Count = 7
Count = 8
Count = 9
Count = 10

This is not what I want. I want what the first instance produces and the second instance to fail. Surely nobody would want otherwise ;-)

Best wishes,
Ahlan

[gautier...@hotmail.com]

Hi Ahlan,

I also get no exception with GNAT GPL 2014, Windows 7...
But with ObjectAda 7.2.2 SE, on the same machine, the 2nd occurrence raises the expected Use_Error.

Cheers
Gautier

NB: ready-to-compile variant below:

with Ada.Text_IO;

procedure Work is
package Io renames Ada.Text_IO;

The_File : Io.File_Type;
begin
Io.Open (The_File, Io.Out_File, "Test.Txt");

for Count in 1..100 loop
Io.Put_Line (The_File, "Count =" & Integer'Image(Count));
Io.Put_Line ("Count =" & Integer'Image(Count));

delay 1.0;
end loop;
Io.Close (The_File);

end Work;

[Björn Lundin]

On 2015-08-27 15:52, ah...@marriott.org wrote:
> Obviously I'm misunderstanding something here.

I'm not sure you do.
On Linux, I start daemons and the open/lock a named file.
if I try to start the daemon again, the locking part
will fail and the second instance will exit.

But I could not solve it in pure Ada.
The Gnat.lockfile package leaves the file if daemon is killed,
so no go using gnat-solution.

I created a small posix package

and used function lockf.
But how to do it on XP, I do not know


--
Björn

[G.B.]

On 27.08.15 15:52, ah...@marriott.org wrote:
> Dear All,
>
> Obviously I'm misunderstanding something here.
>
> I thought that if two processes tried to open the same file for write access then the second process would get an exception.
>
> I have a simple example which when compiled using GnatPro 7.3.1 and run under 32-bit Windows XP demonstrates that this is not so.
>
> Which then begs the question on how to detect if the file is in use and abort if it is.
> This should be simple but obviously too complex for my tiny brain.
> Does anyone know how to do this from Ada (without resorting to the OS directly)?

GNAT RM, section "11.10 Shared Files" addresses this for
GNAT GPL; GNAT Pro should have a similar/the same section.

(The manual is installed with the compiler, e.g. in the
share/doc/gnat subdirectory.)

[Björn Lundin]

On 2015-08-27 16:48, G.B. wrote:
> GNAT RM, section "11.10 Shared Files" addresses this for
> GNAT GPL; GNAT Pro should have a similar/the same section.

Does it ?
To me it looks that it addresses it *within one single process*
Not two different processes opening the same file.


"Section A.14 of the Ada Reference Manual allows implementations to
provide a wide variety of behavior if an attempt is made to access the
same external file with two or more internal files."

and

"When a program that opens multiple files with the same name...."



--
Björn

[ah...@marriott.org]

I've read the manual and (to my reading) it gives the impression that the second opening should raise an exception.
If I try to open the file twice within the same process then it works as expected but NOT if the file is opened twice from different processes.
I can get it to work under Windows if I do my own IO using the windows API directly but of course that is not exactly portable and rather defeats the point of having Ada.Text_Io.
Is this a Gnat bug?
Gautier wrote it works as expected under ObjectAda.

[Jeffrey R. Carter]

On 08/27/2015 08:15 AM, ah...@marriott.org wrote:

> If I try to open the file twice within the same process then it works as
> expected but NOT if the file is opened twice from different processes.

What happens when 2 or more processes try to open the same file for writing is
dependent on the OS. Clearly on Windows there are multiple ways to open a file,
one of which allows multiple processes to open the same file, and this is the
way that GNAT uses.

GNAT would appear to detect multiple openings within the same program in a
manner not dependent on the features of the OS.

The GNAT RM indicates that Create and Open map onto a call to the C function
fopen with certain parameters; you could try creating a C version of your
example using the fopen parameters the RM indicates GNAT uses and see if it
exhibits the same behavior.

The method Ada provides for dealing with such external factors in dealing with
files is the Form parameter to Create and Open. It appears that GNAT does not
provide any way to deal with this situation through the Form parameter. Using
the Form string "shared=no" in your example doesn't affect the behavior.

Another way to get the behavior you're looking for is to use lock files, and
GNAT offers a way to deal with them.

> I can get it to work under Windows if I do my own IO using the windows API
> directly but of course that is not exactly portable and rather defeats the
> point of having Ada.Text_Io.

When neither the language nor the implementation provides a way to get the OS to
do what you want, you have to provide the desired functionality yourself. A
package providing the required operations would have a non-portable body, but
would only require replacing the body to port to a different platform.

> Is this a Gnat bug?

Since the ARM doesn't specify what happens in this case, leaving it up to the
OS, and the OS clearly allows GNAT's behavior, it does not appear to be a GNAT
error. However, since you have GNAT Pro, if you also have a support contract
with AdaCore, you could ask them how to achieve what you want.

--
Jeff Carter
"Have you gone berserk? Can't you see that that man is a ni?"
Blazing Saddles
38

[tmo...@acm.org]

> "Section A.14 of the Ada Reference Manual allows implementations to
> provide a wide variety of behavior if an attempt is made to access the
> same external file with two or more internal files."

Under "Implementation Dependencies" my Janus Ada manual gives the
Form parameters to use to control exclusivity, with the default being
exclusive use.

[Georg Bauhaus]

On 27.08.15 17:08, Björn Lundin wrote:
> On 2015-08-27 16:48, G.B. wrote:
>> GNAT RM, section "11.10 Shared Files" addresses this for
>> GNAT GPL; GNAT Pro should have a similar/the same section.
>
> Does it ?
> To me it looks that it addresses it *within one single process*
> Not two different processes opening the same file.

Uhm, yes. I think that's what Ada internal file objects are about. Given two
or more (instances of) programs, or partitions, and imagining static linking,
still the run-time would otherwise have to "know" the files' full names and
status, system wide, even for NFS, files' aliases…. This seems asking for a lot.

If in Windows a process cannot access a file that is in use,
I think that an effect of the normal reaction of Windows, isn't it?
So, a true Ada for Windows compiler might reflect this OS behavior.

[Pascal Obry]

Le vendredi 28 août 2015 à 01:16 +0200, Georg Bauhaus a écrit :
> If in Windows a process cannot access a file that is in use,
> I think that an effect of the normal reaction of Windows, isn't it?
> So, a true Ada for Windows compiler might reflect this OS behavior.

I think even on Windows a file can be opened with concurrent access. As
already said the behavior is unspecified. Ada RM only talk about
dealing with files in the same executable/process.

BTW, to open a file in shared mode it is possible to use a form string
of Open routine.

--
Pascal Obry / Magny Les Hameaux (78)

The best way to travel is by means of imagination

http://v2p.fr.eu.org
http://www.obry.net

gpg --keyserver keys.gnupg.net --recv-key F949BD3B

[ah...@marriott.org]

Under Windows files may be opened exclusively or shared.
In fact you can have a mixture such as single writer multiple readers.
Multiple writers is possible but rarely used, so it is an odd choice for Gnat to have chosen.
Using form="Shared=no" on the open using Gnat for Windows does not open the file exclusively with regard to other processes.
Which is what I would have expected and I would have thought 99% of users would require.
As far as I can see there is no way that a pure Gnat Ada program can open a file for output exclusively.
If this is true then this is a severe weakness of Gnat.
ObjectAda did what was expected by default.

[Georg Bauhaus]

On 28.08.15 07:41, ah...@marriott.org wrote:
> Under Windows files may be opened exclusively or shared.

No, they can't if using any ISO standard language support? You need OS support from Windows
for lock management, be that with ISO C, or with ISO C++, or with ISO Ada. Using Unix and C,
you will require OS services available through open(2)/flock(2), or some additional library
such as POSIX flockfile(3), etc.. On Windows, with C or Ada, as you have mentioned before,
you need to call some OS functions from Win32, not from ANSI C. The LRMs are not saying things
about the OS or about POSIX or file system here, but about internal Ada file objects in relation
to their names. I think that the GNAT RM is is talking about these, too, not about locking
at external levels. (You may have noticed that Use_Error *is* raised on attempt of using
the same full name in different Ada tasks; these may not be processes.)

So, the name of GNAT's Form parameter, "SHARED" could be misleading: it does not refer
to what "shared" might mean in an OS context.

[Georg Bauhaus]

On 28.08.15 01:29, Pascal Obry wrote:
> Le vendredi 28 août 2015 à 01:16 +0200, Georg Bauhaus a écrit :
>> If in Windows a process cannot access a file that is in use,
>> I think that an effect of the normal reaction of Windows, isn't it?
>> So, a true Ada for Windows compiler might reflect this OS behavior.
>
> I think even on Windows a file can be opened with concurrent access.

Files "can" certainly be opened that way, through OS calls. But the GNAT RM
is, I think, just extending the RM's way of talking about internal file objects
in Ada programs. Is there a language that handles file locking and is *not*
involving external (to the language) services like the OS?

[ah...@marriott.org]

Our working solution was to use the Windows API to open the file and then use Windows API calls to read and write to it.
We did this because using OA there wasn't a way to open files with non-ascii file names.
However it meant that we had to write our own filing system.

Now that we have abandoned OA in favour of Gnat and because Gnat provides a mechanism to open files with non-ascii names I had hoped that we could abandon all this fancy windows stuff and use plain Ada.
The hope was that it would also be platform independent.
From what everyone has written it seems that this is not possible.
It seems that I should somehow lock the file once it has been opened by Ada.
Has anyone any suggestions how best to do that?
Ie How does one write a GNAT Ada program for windows that has exclusive access when opened for write. Surely this must be a common requirement?

On Thursday, August 27, 2015 at 3:52:36 PM UTC+2, ah...@marriott.org wrote:

[Anh Vo]

On Friday, August 28, 2015 at 10:40:47 AM UTC-7, ah...@marriott.org wrote:
> Our working solution was to use the Windows API to open the file and then use Windows API calls to read and write to it.
> We did this because using OA there wasn't a way to open files with non-ascii file names.
> However it meant that we had to write our own filing system.
>
> Now that we have abandoned OA in favour of Gnat and because Gnat provides a mechanism to open files with non-ascii names I had hoped that we could abandon all this fancy windows stuff and use plain Ada.
> The hope was that it would also be platform independent.
> From what everyone has written it seems that this is not possible.
> It seems that I should somehow lock the file once it has been opened by Ada.
> Has anyone any suggestions how best to do that?
> Ie How does one write a GNAT Ada program for windows that has exclusive access when opened for write. Surely this must be a common requirement?

I would suggest to wrap it in a protected object. It checks for task id of a process opening and reading must be the same. Otherwise, the unidentified task will be blocked from accessing the file.

Anh Vo

[Simon Wright]

Anh Vo <anhvo...@gmail.com> writes:

> I would suggest to wrap it in a protected object. It checks for task
> id of a process opening and reading must be the same. Otherwise, the
> unidentified task will be blocked from accessing the file.

That won't help, because the OP wants to protect the file from being
opened by another *process*, not another task, and POs are for managing
access in-process.

As another data point, on OS X it seems that only the output from the
last process to close the file actually appears.

Perhaps GNAT.Lock_Files might help?

[Jeffrey R. Carter]

On 08/28/2015 02:06 PM, Simon Wright wrote:
>
> As another data point, on OS X it seems that only the output from the
> last process to close the file actually appears.

I've also seen this on Linux. Apparently the buffer is big enough to hold the
entire file, and is not written to disk until the file is closed. Since all
programs start writing from the beginning of the file, each subsequent one
overwrites the previous one. If the file were bigger than the buffer, or Flush
were used, one might get different results (especially if the output of the
programs is not all the same).

> Perhaps GNAT.Lock_Files might help?

A lock file will work if all the programs that will access the file will be
written to use the lock file. If you have to protect against any arbitrary
program accessing the file, then you have to use OS features.

--
Jeff Carter
"Propose to an Englishman any principle, or any instrument, however
admirable, and you will observe that the whole effort of the English
mind is directed to find a difficulty, a defect, or an impossibility
in it. If you speak to him of a machine for peeling a potato, he will
pronounce it impossible: if you peel a potato with it before his eyes,
he will declare it useless, because it will not slice a pineapple."
Charles Babbage
92

[Dmitry A. Kazakov]

On Fri, 28 Aug 2015 10:40:45 -0700 (PDT), ah...@marriott.org wrote:

> Now that we have abandoned OA in favour of Gnat and because Gnat provides
> a mechanism to open files with non-ascii names I had hoped that we could
> abandon all this fancy windows stuff and use plain Ada.

I doubt that. Windows "non-ascii" file names are UTF-16. The only
consistent way to have them would be Wide_Wide_Text_IO with names in
Wide_Wide_String internally recoded to UTF-16. Does GNAT this? I didn't
look at the implementation, but I bet it does not. Then how would you do
I/O if the content is not Wide_Wide_String?

> The hope was that it would also be platform independent.

It is Ada standard which prevents reasonable and portable operations with
Unicode file names. You cannot do anything about that.

(I use GIO instead of Ada.Directories when I have to deal with localized
file names.)

> From what everyone has written it seems that this is not possible.

Not if the language I/O abstraction layer does not have this capability.
Ada's does not.

> It seems that I should somehow lock the file once it has been opened by Ada.

Windows locks the file if the sharing mode is 0. See

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx

I don't know what flag GNAT ultimately uses, or if it goes through a POSIX
layer, what the later does. Possibly it ends up with FILE_SHARE_READ.

> Has anyone any suggestions how best to do that?

Use Win32Ada.

> Ie How does one write a GNAT Ada program for windows that has exclusive
> access when opened for write.

By using OS API. If you have do deal with many different OSes create an
abstraction layer and have an implementations of per OS.

> Surely this must be a common requirement?

It is not. Concurrently accessed files presume some persistency being
implemented. It is very uncommon to be done at the file level. DBMS are for
this. When single-file persistency is used it is most of the time an
embedded target where you need not to care about concurrent access.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

[Pascal Obry]

Le samedi 29 août 2015 à 09:05 +0200, Dmitry A. Kazakov a écrit :
> I doubt that. Windows "non-ascii" file names are UTF-16. The only
> consistent way to have them would be Wide_Wide_Text_IO with names in
> Wide_Wide_String internally recoded to UTF-16. Does GNAT this? I
> didn't
> look at the implementation, but I bet it does not. Then how would you
> do
> I/O if the content is not Wide_Wide_String?

I bet you meant Wide_ instead of Wide_Wide_ above. Right?

[Dmitry A. Kazakov]

On Sat, 29 Aug 2015 10:31:56 +0200, Pascal Obry wrote:

> Le samedi 29 août 2015 à 09:05 +0200, Dmitry A. Kazakov a écrit :
>> I doubt that. Windows "non-ascii" file names are UTF-16. The only
>> consistent way to have them would be Wide_Wide_Text_IO with names in
>> Wide_Wide_String internally recoded to UTF-16. Does GNAT this? I
>> didn't
>> look at the implementation, but I bet it does not. Then how would you
>> do
>> I/O if the content is not Wide_Wide_String?
>
> I bet you meant Wide_ instead of Wide_Wide_ above. Right?

No, I meant Wide_Wide_String.

Ada's Wide_String is legally UCS-2. Windows is UTF-16. The only full
Unicode string type is Wide_Wide_String. For an Indo-European language
there is no difference, of course.

Under Linux most applications simply ignore Ada standard and use String
encoded in UTF-8. I suppose that under Linux GNAT calmly passes String file
names as-is, i.e. as UTF-8 [*]. A conformant, but totally useless
implementation would assume names in Latin-1 and recode them into UTF-8
before passing to Linux.

GNAT under Windows is non-conformant either. I doubt it recodes UCS-2
Wide_String into UTF-16. Thus an application that uses Wide_String names
should recode names into UTF-16 first. I.e. same mess as under Linux.

A properly designed Text_IO (Unicode aware) should have used
Wide_Wide_String and/or an UTF-8 encoded string type for all file names
everywhere.

That is why I use GIO instead of Ada standard library. GIO is UTF-8 on both
Windows and Linux, which makes the applications using it portable.

------------------------------------------------------------
* Here is a program illustrating non-conformity of Linux GNAT:

with Ada.Text_IO; use Ada.Text_IO;
with Ada.Characters.Latin_1; use Ada.Characters.Latin_1;

procedure Test_Latin is
File : File_Type;
begin
Create (File, Out_File, "" & LC_A_Diaeresis);
Close (File);
end Test_Latin;

The created file name is garbage, instead of "ä" (a-umlaut).

[gautier...@hotmail.com]

Le samedi 29 août 2015 09:06:00 UTC+2, Dmitry A. Kazakov a écrit :

> On Fri, 28 Aug 2015 10:40:45 -0700 (PDT), ahlan@*** wrote:
>
> > Now that we have abandoned OA in favour of Gnat and because Gnat provides
> > a mechanism to open files with non-ascii names I had hoped that we could
> > abandon all this fancy windows stuff and use plain Ada.
>
> I doubt that. Windows "non-ascii" file names are UTF-16.

Sure, but the form "encoding=utf8" works on GNAT on Windows.
Somewhere in the run-time the UTF-8 name is converted to UTF-16.
So no need of Wide_String or Wide_Wide_String for file names there.
_________________________
Gautier's Ada programming
http://gautiersblog.blogspot.com/search/label/Ada
NB: follow the above link for a valid e-mail address

[Dmitry A. Kazakov]

On Sat, 29 Aug 2015 09:07:07 -0700 (PDT), gautier...@hotmail.com
wrote:

> Le samedi 29 août 2015 09:06:00 UTC+2, Dmitry A. Kazakov a écrit :
>> On Fri, 28 Aug 2015 10:40:45 -0700 (PDT), ahlan@*** wrote:
>>
>>> Now that we have abandoned OA in favour of Gnat and because Gnat provides
>>> a mechanism to open files with non-ascii names I had hoped that we could
>>> abandon all this fancy windows stuff and use plain Ada.
>>
>> I doubt that. Windows "non-ascii" file names are UTF-16.
>
> Sure, but the form "encoding=utf8" works on GNAT on Windows.

Maybe, but this is not Ada.

[Florian Weimer]

* Dmitry A. Kazakov:

> No, I meant Wide_Wide_String.
>
> Ada's Wide_String is legally UCS-2. Windows is UTF-16. The only full
> Unicode string type is Wide_Wide_String. For an Indo-European language
> there is no difference, of course.

GNAT's Wide_String should be compatible with UTF-16. Wide_Wide_String
definitely is not.

> Under Linux most applications simply ignore Ada standard and use String
> encoded in UTF-8. I suppose that under Linux GNAT calmly passes String file
> names as-is, i.e. as UTF-8 [*].

I think GNAT just passes around bytes, it does not care if it's UTF-8
or a legacy encoding. As long as you don't use Wide_String—if you do
use that, things get rather messy.

[Dmitry A. Kazakov]

On Sun, 30 Aug 2015 13:35:16 +0200, Florian Weimer wrote:

> * Dmitry A. Kazakov:
>
>> No, I meant Wide_Wide_String.
>>
>> Ada's Wide_String is legally UCS-2. Windows is UTF-16. The only full
>> Unicode string type is Wide_Wide_String. For an Indo-European language
>> there is no difference, of course.
>
> GNAT's Wide_String should be compatible with UTF-16.

In the sense that Wide_String is an array of 0..2**16-1 units of the
endianness compatible to UTF-16, yes. However the standard is silent about
endianness and alignment (a DSP processor might deploy 32-bit units per
unit)

Semantically no. Wide_String according to RM 3.5.2 (3/3) represents a
narrower set of Unicode than UTF-16.

> Wide_Wide_String definitely is not.

Definitely is. Both are to represent arrays of same sets of Unicode
characters. The representation of Wide_Wide_String is of course irrelevant
when used to specify an external file name, right?

>> Under Linux most applications simply ignore Ada standard and use String
>> encoded in UTF-8. I suppose that under Linux GNAT calmly passes String file
>> names as-is, i.e. as UTF-8 [*].
>
> I think GNAT just passes around bytes, it does not care if it's UTF-8
> or a legacy encoding.

Yes, which is clearly against RM 3.5.2 (2/3) as the example illustrates.
The point is that the behavior one would imply from 3.5.2 and Ada.Text_IO
specification would be useless.

> As long as you don't use Wide_String—if you do use that, things get rather messy.

Both are messy. Character and Ada.Text_IO was designed prior to Unicode.
Later amendments were futile attempts to repair what needed no repair.

[Florian Weimer]

* Dmitry A. Kazakov:

> On Sun, 30 Aug 2015 13:35:16 +0200, Florian Weimer wrote:
>
>> * Dmitry A. Kazakov:
>>
>>> No, I meant Wide_Wide_String.
>>>
>>> Ada's Wide_String is legally UCS-2. Windows is UTF-16. The only full
>>> Unicode string type is Wide_Wide_String. For an Indo-European language
>>> there is no difference, of course.
>>
>> GNAT's Wide_String should be compatible with UTF-16.
>
> In the sense that Wide_String is an array of 0..2**16-1 units of the
> endianness compatible to UTF-16, yes. However the standard is silent about
> endianness and alignment (a DSP processor might deploy 32-bit units per
> unit)
>
> Semantically no. Wide_String according to RM 3.5.2 (3/3) represents a
> narrower set of Unicode than UTF-16.

I don't have a current Windows system to try this, but I think Windows
allows you to use lone surrogates in file names. Such names are not
valid UTF-16, but valid UCS-2. If GNAT supports UCS-2 for
Wide_String, it would actually improve Windows compatibility.

>> Wide_Wide_String definitely is not.
>
> Definitely is. Both are to represent arrays of same sets of Unicode
> characters. The representation of Wide_Wide_String is of course irrelevant
> when used to specify an external file name, right?

You can end up with non-expressible names, depending on how the
conversion to the external representation is performed. (I.e., the
system may have file names which cannot be encoded as
Wide_Wide_String.)

[Dmitry A. Kazakov]

On Sun, 30 Aug 2015 21:37:46 +0200, Florian Weimer wrote:

> * Dmitry A. Kazakov:
>
>> On Sun, 30 Aug 2015 13:35:16 +0200, Florian Weimer wrote:
>>
>>> * Dmitry A. Kazakov:
>>>
>>>> No, I meant Wide_Wide_String.
>>>>
>>>> Ada's Wide_String is legally UCS-2. Windows is UTF-16. The only full
>>>> Unicode string type is Wide_Wide_String. For an Indo-European language
>>>> there is no difference, of course.
>>>
>>> GNAT's Wide_String should be compatible with UTF-16.
>>
>> In the sense that Wide_String is an array of 0..2**16-1 units of the
>> endianness compatible to UTF-16, yes. However the standard is silent about
>> endianness and alignment (a DSP processor might deploy 32-bit units per
>> unit)
>>
>> Semantically no. Wide_String according to RM 3.5.2 (3/3) represents a
>> narrower set of Unicode than UTF-16.
>
> I don't have a current Windows system to try this, but I think Windows
> allows you to use lone surrogates in file names. Such names are not
> valid UTF-16, but valid UCS-2.

The system may have integrated AI that accepts names in English: "a file
with the name of Swahili dhadi". Would it make ASCII same as Unicode?

>>> Wide_Wide_String definitely is not.
>>
>> Definitely is. Both are to represent arrays of same sets of Unicode
>> characters. The representation of Wide_Wide_String is of course irrelevant
>> when used to specify an external file name, right?
>
> You can end up with non-expressible names, depending on how the
> conversion to the external representation is performed. (I.e., the
> system may have file names which cannot be encoded as
> Wide_Wide_String.)

Wasn't the purpose of Unicode to represent all possible characters?

[Florian Weimer]

* Dmitry A. Kazakov:

>>> Semantically no. Wide_String according to RM 3.5.2 (3/3) represents a
>>> narrower set of Unicode than UTF-16.
>>
>> I don't have a current Windows system to try this, but I think Windows
>> allows you to use lone surrogates in file names. Such names are not
>> valid UTF-16, but valid UCS-2.
>
> The system may have integrated AI that accepts names in English: "a file
> with the name of Swahili dhadi". Would it make ASCII same as Unicode?

Sorry, there is no need for being silly. Non-encodable file names are
definite problems and happen in practice (see Java programs on
non-Windows platforms in a multi-byte locale). The user may select a
file, but the application cannot open it. That's a poor user
experience. Or backup software could skip some files because it
cannot seem them, leading to data loss.

>>>> Wide_Wide_String definitely is not.
>>>
>>> Definitely is. Both are to represent arrays of same sets of Unicode
>>> characters. The representation of Wide_Wide_String is of course irrelevant
>>> when used to specify an external file name, right?
>>
>> You can end up with non-expressible names, depending on how the
>> conversion to the external representation is performed. (I.e., the
>> system may have file names which cannot be encoded as
>> Wide_Wide_String.)
>
> Wasn't the purpose of Unicode to represent all possible characters?

This discussion isn't about characters, it's about conversions for
sequences of code units which do not quite match the (current) Unicode
specifications.

[Randy Brukardt]

<gautier...@hotmail.com> wrote in message
news:e908c802-bc4b-4ad7...@googlegroups.com...
> Hi Ahlan,
>
> I also get no exception with GNAT GPL 2014, Windows 7...
> But with ObjectAda 7.2.2 SE, on the same machine, the 2nd occurrence
> raises the expected Use_Error.

Yes, test second occurrence on Janus/Ada raises Use_Error as well.

My recollection is that getting Windows to do this requires opening files
with a particular non-default flag. Probably GNAT does not do that (whether
that is a bug or a feature is not for me to say).

Randy.

[Randy Brukardt]

"Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> wrote in message
news:mvil865iebyb$.1of2shk5faacq$.dlg@40tude.net...
...
>> As long as you don't use Wide_String擁f you do use that, things get

>> rather messy.
>
> Both are messy. Character and Ada.Text_IO was designed prior to Unicode.
> Later amendments were futile attempts to repair what needed no repair.

I would have said: "Later amendments were futile attempts to repair what
needed replacement." since we have no choice but to support modern character
sets in Ada. But the way to do that isn't by abandoning strong typing
(Ada.Strings.UTF_Encoding) or by duplicating everything many times
(Wide_Wide_String, which naturally leads to Wide_Wide_Text_IO which leads to
Wide_Wide_Open which leads to Wide_Wide_Madness).

Since a sensible solution can't be done compatibility, we needed to start
over -- but that's never had appropriate traction.

Randy.

[Dmitry A. Kazakov]

On Mon, 31 Aug 2015 23:12:28 +0200, Florian Weimer wrote:

> * Dmitry A. Kazakov:
>
>>>> Semantically no. Wide_String according to RM 3.5.2 (3/3) represents a
>>>> narrower set of Unicode than UTF-16.
>>>
>>> I don't have a current Windows system to try this, but I think Windows
>>> allows you to use lone surrogates in file names. Such names are not
>>> valid UTF-16, but valid UCS-2.
>>
>> The system may have integrated AI that accepts names in English: "a file
>> with the name of Swahili dhadi". Would it make ASCII same as Unicode?
>
> Sorry, there is no need for being silly.

It is not silly. It is the difference between semantics of the type and a
possibility to misuse bit patterns of type values representation for
anything else. You could put a whole system kernel into a string. That
won't make characters machine instructions.

> Non-encodable file names are
> definite problems and happen in practice (see Java programs on
> non-Windows platforms in a multi-byte locale).

Anything you can encode in Unicode you can encode in Unicode.

> The user may select a
> file, but the application cannot open it. That's a poor user
> experience.

That is not a problem at all. You cannot create a 999TB large file either.
System-specific constraints put on an implementation do not effect the
interface, which has Name_Error in it already.

>>> You can end up with non-expressible names, depending on how the
>>> conversion to the external representation is performed. (I.e., the
>>> system may have file names which cannot be encoded as
>>> Wide_Wide_String.)
>>
>> Wasn't the purpose of Unicode to represent all possible characters?
>
> This discussion isn't about characters, it's about conversions for
> sequences of code units which do not quite match the (current) Unicode
> specifications.

String is an array of characters. Anything that is not a sequence of
characters is not a sequence of characters.

[Dmitry A. Kazakov]

On Mon, 31 Aug 2015 18:34:12 -0500, Randy Brukardt wrote:

> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> wrote in message
> news:mvil865iebyb$.1of2shk5faacq$.dlg@40tude.net...
> ...
>>> As long as you don't use Wide_String擁f you do use that, things get
>>> rather messy.
>>
>> Both are messy. Character and Ada.Text_IO was designed prior to Unicode.
>> Later amendments were futile attempts to repair what needed no repair.
>
> I would have said: "Later amendments were futile attempts to repair what
> needed replacement." since we have no choice but to support modern character
> sets in Ada. But the way to do that isn't by abandoning strong typing
> (Ada.Strings.UTF_Encoding) or by duplicating everything many times
> (Wide_Wide_String, which naturally leads to Wide_Wide_Text_IO which leads to
> Wide_Wide_Open which leads to Wide_Wide_Madness).

Agreed

> Since a sensible solution can't be done compatibility, we needed to start
> over -- but that's never had appropriate traction.

And good so, because without reworking the type system no reasonable
solution is impossible.

Either:

a. Encoding must be an interface, a view, of the string type decoupled from
its internal representation.

b. If the representation is the encoding, then the interface of an array of
characters must still be available.

[brbar...@gmail.com]

When I first read this post, it seemed to me that if the program were
operating sequentially, it would be appropriate to use the Directories
package that already has functions or procedures to detect whether a
file exists and is open. If you use those functions, you can detect
the problem and throw an exception (or some such).

If you're really wanting to do concurrent operations where different
tasks can access an open file, it seems to me that the appropriate
Ada approach is to embed the content of the file in a protected object
and use that capability to avoid the problems with the second file's
access. Chapter 4 in McCormick, Singhoff, and Hugues [Building Parallel,
Embedded, and Real-Time Applications with Ada, 2005, Cambridge U Press]
has most of the material that you would need for this approach.

If you have to deal with unreliable components, then Burns and Wellings
book should have a pretty good start on the algorithms in their "Reader
Writer" example.

You might see if these suggestions would make your work a bit easier.

Bruce B.

[ah...@marriott.org]

I am not trying to protect the file from other tasks I am trying to protect the file from other processes - ie other programs or utilities - perhaps executing on other computers if the file in question is on a network. I want to open the file in such a way that I am granted exclusive write access. This has absolutely nothing to do with tasking or protected objects etc.
Bruce B. tells me that there are routines in the package Directories that can be used to check if the file is already open. However that wouldn't be a full solution because there would be nothing to stop any other program from opening the file and messing with it. However it would be a start. If the file was already opened (by some other process) then I could avoid opening it. Unfortunately I couldn't find any such routine in Ada.Directories. Am I going blind?
So far the only solution I can see is for me to write my own filing system, one that has the ability to open the file for exclusive file access. However I cannot believe that this is what people really do.
As it stands if I write a program that writes a log to a text file a second occurrence of the program if run concurrently will mess up this log.
Is there really no way for an Gnat Ada program to open a file for exclusive access against other processes without resorting to the program writing his/her own filing system? (Open/Read/Write/Close that call directly the OS API)

[Simon Wright]

ah...@marriott.org writes:

> Bruce B. tells me that there are routines in the package Directories
> that can be used to check if the file is already open. However that
> wouldn't be a full solution because there would be nothing to stop any
> other program from opening the file and messing with it. However it
> would be a start. If the file was already opened (by some other
> process) then I could avoid opening it. Unfortunately I couldn't find
> any such routine in Ada.Directories. Am I going blind?

GNAT.Directories perhaps? though I only see an Is_Open for directories,
not files.

GNAT.OS_Lib (a renaming of System.OS_Lib)? no, don't think so.

> So far the only solution I can see is for me to write my own filing
> system, one that has the ability to open the file for exclusive file
> access. However I cannot believe that this is what people really do.
> As it stands if I write a program that writes a log to a text file a
> second occurrence of the program if run concurrently will mess up this
> log.
> Is there really no way for an Gnat Ada program to open a file for
> exclusive access against other processes without resorting to the
> program writing his/her own filing system? (Open/Read/Write/Close that
> call directly the OS API)

I misunderstood you. To me, "filing system" means the whole organisation
of files with hierarchies of directories.

I think you are rather out of luck. I can see how it might be done on OS
X [open(2) with O_EXLOCK, then write(2)].

But, if you want a logger, what about for example Alog[1]? It's probably
overkill. But it's an exact analog of how you'd do it for tasking within
one process: open the file in a dedicated task, and send messages to be
logged to the task.

[1] http://www.codelabs.ch/alog/

[Jacob Sparre Andersen]

Ahlan Marriott <ah...@marriott.org> wrote:

> I am not trying to protect the file from other tasks I am trying to
> protect the file from other processes - ie other programs or utilities
> - perhaps executing on other computers if the file in question is on a
> network. I want to open the file in such a way that I am granted
> exclusive write access.

I'm not even sure all operating systems support this.

On Unix systems you typically see programs creating a separate "lock
file" (containing the process ID of the creator) for each locked file.

Greetings,

Jacob
--
"When we cite authors we cite their demonstrations, not their names"
-- Pascal

[ah...@marriott.org]

I have no idea about Linux but it is perfectly possible under Windows. However this doesn't stop people (including Microsoft) from creating lock files that then sometimes get left behind when a program crashes rendering the file locked forever! So the presence of lock files doesn't mean that the opsys can't lock exclusively.
I mentioned a logger not because I want to write a logger but as a simple example of somthing someone might want to write.

[G.B.]

On 01.09.15 16:05, ah...@marriott.org wrote:
> So far the only solution I can see is for me to write my own filing system, one that has the ability to open the file for exclusive file access. However I cannot believe that this is what people really do.

If you mean system wide exclusive access to operating system
resources like files needing operating system calls, or else
system defaults, everybody does it that way all the time, I think.

Similarly, access control (users, groups, roles) isn't portable
either, so just calling some ISO language's "file open" with
a Form parameter might or might not be sufficient.

If some tentative call, were it to ever be successful, has

Name => "//Devices/IO/Network/Card/0",
Form => "Exclusive"

the effects you mentioned WRT failure, or even just on timeouts,
might be more striking ...

[Pascal Obry]

Le lundi 31 août 2015 à 18:20 -0500, Randy Brukardt a écrit :
> My recollection is that getting Windows to do this requires opening
> files
> with a particular non-default flag. Probably GNAT does not do that
> (whether
> that is a bug or a feature is not for me to say).

If my recollection is correct this is undefined by the language, so it
can't be a bug.

[brbar...@gmail.com]

My Linux version of the reference manual is a bit hard to find from my e-mail.
The Ada-2012 version of the RM has a function Is_Open that returns a Boolean
for each of the IO types defined in that document. For example section A.10.1
has Is_Open as the first function in the description for Text_IO. I apologize
for not checking Ada.Directories in the RM when I posted my note earlier today.

I'm still not sure why the alternate processes you're referring to wouldn't
be Ada Tasks. In that case, the appropriate design would embed the information
you want to protect into an object that forces the other tasks to wait until
the task accessing the file ends. If you are really running multi-language
implementations, you'll probably need to uncover the code to identify which
process is your Ada process and then have code that interfaces with the Ada.

If you try to run concurrent write operations to text files without using
protected objects or some other form of locking, the IO buffer will certainly
get confused. As a result, the output will have a strange interleaving of
characters from the different processes. This difficulty certainly makes
debugging by printing lines of text more complicated.

McCormick, et al. have a
very nice (and simple) code example for a package called "Protected_Output"
that uses a pattern called a Semaphore that deals with this problem. The
code is on pp. 145-146 of their book. It may be that their web site
<htttp://www.cambridge.org/9780521197168> can provide the source code.
After you open the site, find a tab labelled "Resources". Then choose
one called "Source Code" and download the zip file. The Protected_Output
code is from section 4.7 in the book. I think the driver for this code
is included in the zip file.

I have compiled and run this demo successfully. I strongly suggest you consult
McCormick et al. before you try writing your own file system.

I do apologize for not carefully checking the RM for the proper location
and name of the Is_Open function.

Bruce B.

[Randy Brukardt]

<ah...@marriott.org> wrote in message
news:ed3129f2-7c95-4ad9...@googlegroups.com...

On Thursday, August 27, 2015 at 3:52:36 PM UTC+2, ah...@marriott.org wrote:

...

>So far the only solution I can see is for me to write my own filing system,
>one
>that has the ability to open the file for exclusive file access. However I
>cannot
>believe that this is what people really do.

Well, obviously one could use a different compiler, since it appears that
most Windows compilers do this right (we know for a fact that ObjectAda and
Janus/Ada do so). Probably this is a side-effect of mapping a Unx-derived
compiler onto Windows.

>Is there really no way for an Gnat Ada program to open a file for exclusive
>access against other processes without resorting to the program writing
>his/her
>own filing system? (Open/Read/Write/Close that call directly the OS API)

Dunno. In any case, this is implementation-specific stuff (usually involving
a Form parameter), so there is not going to be a portable solution. So you
have to contact your vendor for help. Of course, if you don't have support
you'll be SOL, but you get want you pay for. ;-)

Randy.

[Randy Brukardt]

"Simon Wright" <si...@pushface.org> wrote in message
news:lyh9nex...@pushface.org...

> ah...@marriott.org writes:
>
>> Bruce B. tells me that there are routines in the package Directories
>> that can be used to check if the file is already open. However that
>> wouldn't be a full solution because there would be nothing to stop any
>> other program from opening the file and messing with it. However it
>> would be a start. If the file was already opened (by some other
>> process) then I could avoid opening it. Unfortunately I couldn't find
>> any such routine in Ada.Directories. Am I going blind?

...

> I think you are rather out of luck. I can see how it might be done on OS
> X [open(2) with O_EXLOCK, then write(2)].

It's trivial to do on Windows, one just needs to open the file with the
correct sharing flags (and I think that the correct flag is the default).
Sharing is a built-in part of the file descriptor in Windows; there's no
need for separate locking so long as you want to do full-file exclusion (as
in this case).

Typically, those sharing flags are made available through a Form parameter.
Janus/Ada uses 'R' and 'W' to indicate read and write sharing
(respectively). (So a form of "RW" opens a file allowing read and write
sharing, while opening a file with "" - the default - opens it for exclusive
access.) But this will be different on different compilers, plus it's
specific to Windows.

Randy.

[Randy Brukardt]

"Jacob Sparre Andersen" <spa...@nbi.dk> wrote in message
news:87d1y22...@adaheads.sparre-andersen.dk...

> Ahlan Marriott <ah...@marriott.org> wrote:
>
>> I am not trying to protect the file from other tasks I am trying to
>> protect the file from other processes - ie other programs or utilities
>> - perhaps executing on other computers if the file in question is on a
>> network. I want to open the file in such a way that I am granted
>> exclusive write access.
>
> I'm not even sure all operating systems support this.

Most don't. That's why it's implementation-defined, and you're SOL if your
implementation doesn't properly implement the Windows sharing flags (which
have no Unix counterpart).

Randy.

[Randy Brukardt]

"Pascal Obry" <pas...@obry.net> wrote in message
news:1441124612.2...@obry.net...

Le lundi 31 août 2015 à 18:20 -0500, Randy Brukardt a écrit :
>> My recollection is that getting Windows to do this requires opening files
>> with a particular non-default flag. Probably GNAT does not do that
>> (whether
>> that is a bug or a feature is not for me to say).
>
>If my recollection is correct this is undefined by the language, so it
>can't be a bug.

It can be a bug if GNAT intended to make the Windows sharing flags available
via a Form parameter, and just got it wrong somehow. It might be a feature
if GNAT's authors were trying to make Windows look like Linux (a serious
mistake IMHO, but as you say a legitimate choice vis-a-vis the RM).

I would have expected the former since AdaCore doesn't usually make easy
things impossible for their customers. They claim (and it is backed up by
practice) to be customer-centric. I'd expect that this capability would be
available if some GNAT Pro customer had reported the problem, so perhaps it
is not as important as it seems.

My personal experience is that I use the sharing flags a lot. For instance,
for loggers, I usually open the files allowing read sharing (but not write
sharing) so that I can read the log file while the service is running.
(Exclusive access is exactly that, meaning that you can't even read the log
file while it is open, and that can be pain for long-running programs.) For
most other sorts of files, exclusive access is best. The Unix strategy of
allowing anything is obnoxious.

Randy.

[Simon Wright]

brbar...@gmail.com writes:

> I'm still not sure why the alternate processes you're referring to
> wouldn't be Ada Tasks.

Because they aren't?

If in one terminal I say "./a_program" then a_program is a process
running under the OS. It may contain many Ada tasks. If in another
terminal I say "./another_program" then another_program is another
process running under the OS, which may contain a whole lot of Ada
tasks, which know nothing about a_program's tasks. OP wants to ensure
exclusive access to a file between the two processes.

[Björn Lundin]

Or it can be notepad.exe ...
Or anything else that is NOT a process written in ada

--
--
Björn

[Björn Lundin]

On 2015-09-01 17:31, ah...@marriott.org wrote:

> I have no idea about Linux but it is perfectly possible under Windows.
> However this doesn't stop people (including Microsoft) from creating lock
> files that then sometimes get left behind when a program crashes
> rendering the file locked forever!
> So the presence of lock files doesn't mean that the opsys can't lock
exclusively.

That is why you do not want to use gnat.lockfiles, (at least on linux)
but instead use somthing like linux' lockf.

If the process holding the lock gets killed, (even by kill -kill)
the kernel (or something) releases the lock.
No stale lockfiles.


--
Björn

[brbar...@gmail.com]

I have been debugging a concurrent Ada program (on Windows XP, but I
expect the GNAT GPL would work the same on Linux). In the GPS debugger,
there is a task window. The primary procedure appears as a task, which I
believe is going to be related to the thread for the process. If I run a
strictly sequential version of the program there's only one task that appears
in the debugger window for tasks. When I switch over the the concurrent
version (which I'm still debugging), then I can have the program create,
run, and terminate a large number of tasks. These appear in the task
window including their state (runnable, ..., terminated). [I'll provide
the code after I'm satisfied with the debugging process.]

I believe the debugger provides a strong prima facie case the (at least
for GNAT GPL), Ada will treat a single process as a task. When you add
subroutines in other languages to this code, it seems likely that the
Ada compiler would still treat the main procedure as a task - although I do
not have direct evidence of that.

To the extent that the Ada program introduces other tasks, the program
falls into the realm of concurrent programming, where the appropriate
Ada approach is to deal with resource starvation or blocking with protected
objects or some of the more primitive approaches (such as semaphores).

If the program is going to run in an environment that involves multiple
machines, then in addition to concurrent programming, the system design
will involve distributed computing. Because multi-system designs usually
have to assume that the individual nodes are unreliable, the algorithms
become more complex because of the need to check on whether the components
are unavailable. That would be the case, for example, in dealing with
a web site that uses clusters of servers and storage units.

Bruce B.

[Björn Lundin]

On 2015-09-06 19:38, brbar...@gmail.com wrote:
> On Saturday, September 5, 2015 at 11:56:28 AM UTC-4, björn lundin wrote:
>>
>>> I have no idea about Linux but it is perfectly possible under Windows.
>>> However this doesn't stop people (including Microsoft) from creating lock
>>> files that then sometimes get left behind when a program crashes
>>> rendering the file locked forever!
>>> So the presence of lock files doesn't mean that the opsys can't lock
>> exclusively.
>>
>> That is why you do not want to use gnat.lockfiles, (at least on linux)
>> but instead use somthing like linux' lockf.
>>
>> If the process holding the lock gets killed, (even by kill -kill)
>> the kernel (or something) releases the lock.
>> No stale lockfiles.
>>
>>
>> --
>> Björn
>
> I have been debugging a concurrent Ada program (on Windows XP, but I
> expect the GNAT GPL would work the same on Linux).

We are talking about different things.
The op said he had *several processes*. Not one process with several
tasks. My posting above is directed to the case with several processes,
not several tasks.

Say you have a program that should run almost all the time.
But when it experiences external trouble (network down), its easier to
just exit the program, and have a scheduler to restart it.
(crude but simple and good enough in my case)

Now, the scheduler needs find out if one instance is running
or not. More than one instance is no good.

The program then uses lockf to lock a file exclusively at startup.
The scheduler runs once a minute and tries to lock that same file.
If failure - the process runs and that is it.
If success - release the lock, and restart the process.

Also, if the process is restarted by mistake, it
will fail to lock the file, and exit.
There is already an instance running.

Has nothing to do with tasks.
Its like opening the file with notepad and word at the same time.


--
--
Björn

[brbar...@gmail.com]

I understand the context of your thinking. My comment was specifically
directed at separating contexts dealing with Ada approaches to the problem.
There's a context in which there's only a sequential Ada program, another
in which the Ada program involves several tasks, possibly including
interactions with processes under the control of the OS on a single
machine. There's a third involving processes running on different machines.
The context you're assuming appears to fall into the second or third of these.
The second context may involve what's usually called "low level" interactions
with the OS, possibly using mutexes or other library procedures. From an Ada
perspective the third may require items discussed in the Distributed System
Annex.

Bruce B.

[Florian Weimer]

* Dmitry A. Kazakov:

> Anything you can encode in Unicode you can encode in Unicode.

This is simply not true across multiple Unicode versions (and when
dealing with multi-byte encodings). It is not possible to express all
possible UCS-2 sequences in UTF-32, in the sense that converting from
UTF-32 completely covers the entire set of UCS-2 strings.

>> The user may select a file, but the application cannot open it.
>> That's a poor user experience.
>
> That is not a problem at all. You cannot create a 999TB large file either.

Well ...

-rw-r--r-- 1 fw fw 1098412116148224 Sep 7 20:23 t

And I expect that I can delete it, too. Anything else would be a bug.

> System-specific constraints put on an implementation do not effect the
> interface, which has Name_Error in it already.

It's not a constraint that the system imposes, it's something that
requires careful thought from the programming language implementor.
If they impose stronger constraints on strings than what the file
system enforces, you end up with non-nameable files and a poor user
experience.

[Dmitry A. Kazakov]

On Mon, 07 Sep 2015 20:27:58 +0200, Florian Weimer wrote:

> * Dmitry A. Kazakov:
>
>> Anything you can encode in Unicode you can encode in Unicode.
>
> This is simply not true across multiple Unicode versions (and when
> dealing with multi-byte encodings).

Versions of Unicode are irrelevant.

> It is not possible to express all
> possible UCS-2 sequences in UTF-32, in the sense that converting from
> UTF-32 completely covers the entire set of UCS-2 strings.

You can represent all code ranges in UTF-32. I don't think the language
should care about "visible characters." This insanity can be safely
ignored.

>>> The user may select a file, but the application cannot open it.
>>> That's a poor user experience.
>>
>> That is not a problem at all. You cannot create a 999TB large file either.
>
> Well ...
>
> -rw-r--r-- 1 fw fw 1098412116148224 Sep 7 20:23 t
>
> And I expect that I can delete it, too. Anything else would be a bug.

Bug of the OS or of your program?

Ada.Text_IO could also tell you that the file was created and then proclaim
anything else a bug.

>> System-specific constraints put on an implementation do not effect the
>> interface, which has Name_Error in it already.
>
> It's not a constraint that the system imposes, it's something that
> requires careful thought from the programming language implementor.

You cannot have files named ".." on some OSes. That has nothing to do with
Ada or any other language.

> If they impose stronger constraints on strings than what the file
> system enforces, you end up with non-nameable files and a poor user
> experience.

That is the problem with current Ada.Directories and Ada.Text_IO. Not
because of a constraint, but because the constraint is meaningless in a
system that allows wider sets of characters than ASCII.

[Florian Weimer]

* Dmitry A. Kazakov:

> On Mon, 07 Sep 2015 20:27:58 +0200, Florian Weimer wrote:
>
>> * Dmitry A. Kazakov:
>>
>>> Anything you can encode in Unicode you can encode in Unicode.
>>
>> This is simply not true across multiple Unicode versions (and when
>> dealing with multi-byte encodings).
>
> Versions of Unicode are irrelevant.

Not true.

>> It is not possible to express all
>> possible UCS-2 sequences in UTF-32, in the sense that converting from
>> UTF-32 completely covers the entire set of UCS-2 strings.
>
> You can represent all code ranges in UTF-32. I don't think the language
> should care about "visible characters." This insanity can be safely
> ignored.

Not true. Please read the specification and the restrictions on the
surrogate range.