Windows env-Bug?
Gerhard Reithofer
I've found a strange behaviour in Windows which I would call a bug:
-----------------------------------------
% set env(Test) "Value"
Value
% info exists env(Test)
1
% puts $env(Test)
Value
% set env(Test) ""
% info exists env(Test)
1
% puts $env(Test)
can't read "env(Test)": no such variable
-----------------------------------------
[info exists ..] returns 1 but the variable cannot be dereferenced.
% puts $tcl_patchLevel
8.4.13
% parray tcl_platform
tcl_platform(byteOrder) = littleEndian
tcl_platform(machine) = intel
tcl_platform(os) = Windows NT
tcl_platform(osVersion) = 5.0
tcl_platform(platform) = windows
Same on Windows XP (Windows NT 5.1) with TCL 8.4.11,
no problem on Linux and Unix (AIX), "" is returned.
Has anyone else seen this behaviour before?
Does one have a good idea as workaround?
Controlling the environment is essential to a project where I'm working
on.
--
Gerhard Reithofer
Tech-EDV Support Forum - http://support.tech-edv.co.at
Torsten Edler
windows prompt
set XX=AA
is setting the variable to value AA. can be checked by using
set
and unsetting the variable can be done by
set XX=
which is setting the variable to empty string - and in windows this
means deleting the variable! Again another check will confirm that the
variable XX is gone. Same can be done in TCL via the env array and
using parray.
Helmut Giese
While this is true, the fact that
set env(Test) ""
info exists env(Test)
returns '1' is certainly unfortunate.
It seems that
set env(Test) ""
set Tcl's copy of 'env' to "" - which is perfectly legal for a Tcl
array, thus causing [info exists] to return '1' - but when trying to
retrieve this supposedly existing element, Tcl seems to go out asking
Windows for the value - and for Windows it doesn't exist any more.
Probably it would be quite messy to fix this.
Just my 0.02
Helmut Giese
Torsten Edler
Still stranger: when you do an "array names env" before "info exists
env(Test)" it vanishes in TCL.
(bin) 38 % info patchlevel
8.4.13
(bin) 39 % set env(Test) "X"
X
(bin) 40 % info exists env(Test)
1
(bin) 41 % set env(Test) ""
(bin) 42 % info exists env(Test)
1
(bin) 43 % array names env
HOME PROCESSOR_IDENTIFIER ComSpec DEVMGR_SHOW_DETAILS
DEVMGR_SHOW_NONPRESENT_DEVICES JSERV HOMEDRIVE ALLUSERSPROFILE
FP_NO_HOST_CHECK SESSIONNAME PROCESSOR_REVISION PATHEXT LOGONSERVER
VS80COMNTOOLS USERDOMAIN PROCESSOR_LEVEL DEFAULT_CA_NR USERPROFILE
USERNAME COMPUTERNAME TREECTRL_LIBRARY TMP SystemRoot
PROCESSOR_ARCHITECTURE OS ProgramFiles APPDATA CommonProgramFiles
WV_GATEWAY_CFG windir Path TEMP HOMEPATH VU_LIBRARY SystemDrive
PERL5LIB NUMBER_OF_PROCESSORS
(bin) 44 % info exists env(Test)
0
(bin) 45 %
Helmut Giese
So it seems that [array names] does some magic when the array in
question is 'env' - namely unsetting the internal 'env' array and then
re-populating it by asking Windows for all existing environment
variables - et voilà they're both in sync again.
Helmut Giese
Gerhard Reithofer
> On 21 Jul., 20:33, Helmut Giese <hgi...@ratiosoft.com> wrote:
> > On Sat, 21 Jul 2007 13:31:04 -0000, Torsten Edler
...
> > <Torsten.Ed...@googlemail.com> wrote:
> > >This not a tcl feature - this is a windows feature. Try this at the
...
> Still stranger: when you do an "array names env" before "info exists
> env(Test)" it vanishes in TCL.
...
Thanks, this is the usable workaround which I missed.
Nevertheless the TCL behaviour is IMHO a bug. Should I report a bug to
Sourceforge or does someone already feel responsible to work on that?
PS: A real great community the c.l.t - THX
Helmut Giese
<gerhard....@tech-edv.co.at> wrote:
>Nevertheless the TCL behaviour is IMHO a bug. Should I report a bug to
>Sourceforge or does someone already feel responsible to work on that?
It's surely safest to do it yourself.
Maybe you want to add the content of this thread, since it contains
some observations which might be helpful.
Best regards
Helmut Giese
Gerhard Reithofer
Done - request ID 1758527
David Gravereaux
> Nevertheless the TCL behaviour is IMHO a bug. Should I report a bug to
> Sourceforge or does someone already feel responsible to work on that?
Forcing a re-ask of the environment for any access to the array would hurt
performance. Maybe a special behavior regarding setting any variable to ""
implies a delete (all platforms?), but the environment handling code
doesn't have platform specific code to it.
If you want to point a finger, point it at libc's setenv() that Tcl uses.
Yes, it's true that Tcl's behavior is not consistent on windows, but that
is the behavior on that platform. What about the next person who wonders
why Tcl can't set an envvar to empty (do we use the no-break space char to
imply empty?), then comes another person who wonders why their DLL changing
an envvar doesn't cause Tcl to fire trace on it or even show it be the same.
^- hint: The real environment is not sync'd to libc's cache of it.
In a way, this is like exit codes. Win apps have a 32-bit exit code. Tcl
only does 7-bit. Do you raise Tcl's size or truncate it to fit? Well, in
our case, both are happening and neither is fully right.
Patch it if you can, but you'll be staring at best a compromise that might
not address each side effectively.
--
"MOM, CAN I SET FIRE TO MY BED MATTRESS?" "No, Calvin." "CAN I RIDE MY
TRICYCLE ON THE ROOF?" "No, Calvin." "Then can I have a cookie?" "No, Calvin."
("She's on to me.")
Gerhard Reithofer
> Gerhard Reithofer wrote:
>
> > Nevertheless the TCL behaviour is IMHO a bug. Should I report a bug to
> > Sourceforge or does someone already feel responsible to work on that?
>
> Forcing a re-ask of the environment for any access to the array would hurt
> performance.
Of cource, but it seems to be a workaround for getting the correct
value.
> Maybe a special behavior regarding setting any variable to ""
> implies a delete (all platforms?), but the environment handling code
> doesn't have platform specific code to it.
>
> If you want to point a finger, point it at libc's setenv() that Tcl uses.
>
> Yes, it's true that Tcl's behavior is not consistent on windows, but that
> is the behavior on that platform. What about the next person who wonders
> why Tcl can't set an envvar to empty (do we use the no-break space char to
> imply empty?), then comes another person who wonders why their DLL changing
> an envvar doesn't cause Tcl to fire trace on it or even show it be the same.
> ^- hint: The real environment is not sync'd to libc's cache of it.
>
> In a way, this is like exit codes. Win apps have a 32-bit exit code. Tcl
> only does 7-bit. Do you raise Tcl's size or truncate it to fit? Well, in
> our case, both are happening and neither is fully right.
>
> Patch it if you can, but you'll be staring at best a compromise that might
> not address each side effectively.
I only complained that the $env array does not reflect the state of the
system environment - nothing more was requested from my side.
The inconsistence between [info exists ...] and a none-existing value
makes the handling of this situation even more cumbersome.
suchenwi
edv.co.at> wrote:
> I only complained that the $env array does not reflect the state of the
> system environment - nothing more was requested from my side.
> The inconsistence between [info exists ...] and a none-existing value
> makes the handling of this situation even more cumbersome.
As another workaround, you might also use
if ![catch {set ::env(MyTest)}] ...
instead of
if [info exists ::env(MyTest)] ...
to prevent rescanning the whole environment.
Gerhard Reithofer
Nice idea, thanks.
Donal K. Fellows
> Maybe a special behavior regarding setting any variable to ""
> implies a delete (all platforms?),
Categorically not; there are a number of UNIX environment variables
that have a different meaning when set to the empty string as opposed
to when unset. One of the root problems here is that the UNIX and
Windows behaviours are just different here; it has the feel of a short-
cut taken long ago (in MS-DOS 1.0 or 2.0!) that's now too deeply
ingrained to fix. :-\
On the other hand, that doesn't mean that we shouldn't add some code
so that the unsetting of the element happens immediately on Windows.
But it's definitely going to be platform-specific because the
underlying environment variable behaviour is also platform-specific.
Can't be helped really.
Donal.
David Gravereaux
> David Gravereaux wrote:
>> Maybe a special behavior regarding setting any variable to ""
>> implies a delete (all platforms?),
>
> Categorically not; there are a number of UNIX environment variables
> that have a different meaning when set to the empty string as opposed
> to when unset.
I'm not serious.. just rattling chains.
> One of the root problems here is that the UNIX and
> Windows behaviours are just different here; it has the feel of a short-
> cut taken long ago (in MS-DOS 1.0 or 2.0!) that's now too deeply
> ingrained to fix. :-\
>
> On the other hand, that doesn't mean that we shouldn't add some code
> so that the unsetting of the element happens immediately on Windows.
> But it's definitely going to be platform-specific because the
> underlying environment variable behaviour is also platform-specific.
> Can't be helped really.
The code for it is all platform neutral libc. It'll take some work to split
it out. The caching behavior might have to be removed which would render
traces inoperative.
Currently, a DLL (in the chain of things) might call win32's
SetEnvironmentVariable() which is not reflected by libc's cache of the real
environment which Tcl sees. Tcl goes so far as to export a Tcl_PutEnv (i
think) so traces can fire, but is meaningless to code that uses the real win32
system environment access functions.
Who weighs more there? The real system environment to obtain correct answers,
or libc's buggy behavior due to system differences which has the ability to
fire traces?
Here's that compromise part in all its ugliness.
--
"Someday I'll write my own philosophy book." -Calvin
Joe English
>
>Currently, a DLL (in the chain of things) might call win32's
>SetEnvironmentVariable() which is not reflected by libc's cache of the real
>environment which Tcl sees. Tcl goes so far as to export a Tcl_PutEnv (i
>think) so traces can fire, but is meaningless to code that uses the real win32
>system environment access functions.
>
>Who weighs more there? The real system environment to obtain correct answers,
>or libc's buggy behavior due to system differences which has the ability to
>fire traces?
>
>Here's that compromise part in all its ugliness.
That's another reason I think Tcl's treatment of ::env
ought to be changed. Instead of trying (and often failing)
to keep ::env in sync with the system environment at all times,
Tcl could initialize ::env from the system environment
at program startup only.
Slave interps and new threads would inherit ::env from their
parent interp, and the current interp's ::env would be used
to initialize the system environment for child processes
spawned via [exec], [open "|..."], and the like.
Decoupling ::env from the system environment after initialization
would also solve a number of thread-safety problems.
--Joe English
Gerhard Reithofer
> David Gravereaux wrote:
> >
> >Currently, a DLL (in the chain of things) might call win32's
> >SetEnvironmentVariable() which is not reflected by libc's cache of the real
...
> That's another reason I think Tcl's treatment of ::env
> ought to be changed. Instead of trying (and often failing)
> to keep ::env in sync with the system environment at all times,
> Tcl could initialize ::env from the system environment
> at program startup only.
>
> Slave interps and new threads would inherit ::env from their
> parent interp, and the current interp's ::env would be used
> to initialize the system environment for child processes
> spawned via [exec], [open "|..."], and the like.
>
> Decoupling ::env from the system environment after initialization
> would also solve a number of thread-safety problems.
What do you mean by "Decoupling"?
Controlling (and not "monitoring") the process environment is one of the
basic methods of the "interprocess communication", its one way to
forward informations to subprocesses and change information on current
process level.
The "current process level" arguments is discussed controversly, I don't
know a standard for that as the $env mechanims has been created very
long before threads hav been introduced.
In my specific situation I have to create more that 100 environment
variables which are required by the subprocess (configuration options).
One specific reason for using Tcl is, that some unix tools (awk, sed,
...) fail by working too long variable values and I think this may also
be a problem under windows in some situations (eg. very a log PATH
variable?).
Tcl has no problem with very long lists, variable contents ... an also
allows platform indepenent path handling.
Thus it is the perfect platform independent "configuration machine" in
this situation and allows very convenient and efficient "environment
handling" for subprocesses.
But it must be ensured, that the internal state of ::env variable
reflect the platform specific environment of the subprocess.
One more reason to work on ::env is the fact, that windows treats
environment variables case-insensitive, unix does not:
Unix:
% set env(a) 1
1
% set env(A) 2
2
% puts "$env(a) $env(A)"
1 2
Windows:
% set env(a) 1
1
% set env(A) 2
2
% puts "$env(a) $env(A)"
2 2
parray only lists env(a) (the 1st one).
Alexandre Ferrieux
edv.co.at> wrote:
>
> Controlling (and not "monitoring") the process environment is one of the
> basic methods of the "interprocess communication", its one way to
> forward informations to subprocesses and change information on current
> process level.
It's the most basic method, yes: unidirectional (from parent to
child), and one-shot (only at spawn time).
But this part is not threatened by Joe's proposal: on the contrary, he
suggests to keep only this part (fill ::env with tclsh's environment
at interp init time, and update the environment of children based
on ::env's contents, only at spawn time; nothing in between. Hence no
cpu-hungry traces on ::env).
I for one am in favour of Joe's idea: for an unixian it sounded a bit
alien anyway to see the environment change in the middle of the life
of a process...
(Now it remains to be checked whether some popular extension will veto
this move because it spawns a process without using the Tcl API,
making it hard to apply this delay-update scheme)
-Alex
Gerhard Reithofer
> On Jul 28, 11:28 am, Gerhard Reithofer <gerhard.reitho...@tech-
> edv.co.at> wrote:
> >
> > Controlling (and not "monitoring") the process environment is one of the
> > basic methods of the "interprocess communication", its one way to
> > forward informations to subprocesses and change information on current
> > process level.
>
> It's the most basic method, yes: unidirectional (from parent to
> child), and one-shot (only at spawn time).
>
> But this part is not threatened by Joe's proposal: on the contrary, he
> suggests to keep only this part (fill ::env with tclsh's environment
> at interp init time, and update the environment of children based
> on ::env's contents, only at spawn time; nothing in between. Hence no
> cpu-hungry traces on ::env).
I'm aware that Joe's proposal was defined that way, but my problem is
that I don't have a feeback what really happens in the spawned process
if I do not reread the enviroment, therefore I cannot "control" it.
Example using windows case-insensitive env:
% set env(Hello) hello
hello
% puts $env(hELLo)
hello
% info exists env(HeLlo)
1
% set env(hElLo) ""
% info exists env(HeLlo)
1
% puts $env(HeLlo)
can't read "env(HeLlo)": no such variable
So it's hard to determine the exact situation of a subprocess. Maybe one
solution may be to dissociate the ::env array (which could be a fast
access method for "process inititial" situation) and on the other way
methods for reliable setting and getting the "real evironment" values,
but this would require new platform specific functions I think.
> I for one am in favour of Joe's idea: for an unixian it sounded a bit
> alien anyway to see the environment change in the middle of the life
> of a process...
This is the basic job of a shell or command interpreter to change the
currrent environment and start a sub process, change it again...
> (Now it remains to be checked whether some popular extension will veto
> this move because it spawns a process without using the Tcl API,
> making it hard to apply this delay-update scheme)
Interesting point.
IMHO Tcl is a "shell" and should also provide a mechanism for
"controlling" the environement ;-)
BTW: I didn't find a Tcl library call for executing a subprocess - have
I overseen something?
Helmut Giese
<gerhard....@tech-edv.co.at> wrote:
Hi Gerald,
>I'm aware that Joe's proposal was defined that way, but my problem is
>that I don't have a feeback what really happens in the spawned process
>if I do not reread the enviroment, therefore I cannot "control" it.
AFAIK this won't work because the environment is a "one way street":
You can set it up any way you want and spawn a child process and this
process might change it - but you / the parent won't be able to see
these changes. If you re-read the environment, you will ever only get
your own.
At least that's what I learned and I have never seen any different
setup.
Best regards
Helmut Giese
Alexandre Ferrieux
edv.co.at> wrote:
>
> I'm aware that Joe's proposal was defined that way, but my problem is
> that I don't have a feeback what really happens in the spawned process
> if I do not reread the enviroment, therefore I cannot "control" it.
You've just discovered that the environment is not really a means of
interprocess communication.
One of Tcl's nice features is some level of platform abstraction.
Since on unix, the environment works only in the lightweight way
described before, then if you want to stay portable you cannot do
more. Now if you're okay to do something Windows-specific, then by all
means do, with specific functions (guess these are already in TWAPI).
> > I for one am in favour of Joe's idea: for an unixian it sounded a bit
> > alien anyway to see the environment change in the middle of the life
> > of a process...
>
> This is the basic job of a shell or command interpreter to change the
> currrent environment and start a sub process, change it again...
No. I mean "in the middle of the child's life".
> > (Now it remains to be checked whether some popular extension will veto
> > this move because it spawns a process without using the Tcl API,
> > making it hard to apply this delay-update scheme)
>
> Interesting point.
> IMHO Tcl is a "shell" and should also provide a mechanism for
> "controlling" the environement ;-)
Again, on unix you cannot do anything to the env of an already running
child.
Nor would anybody want to anyway (there *are* true IPC methods).
Hence, when a shell or another program modifies its own environment,
it is updating only an internal data structure like any other, which
will take special meaning only on next spawn. Hence Joe's proposal to
let ::env play this role (just like any other global array), and only
intercept spawns, at which time ::env should be read to generate the
'environ' argument of the execve() (or equivalent) call. Hence no need
to cripple ::env with read/write traces. QED.
> BTW: I didn't find a Tcl library call for executing a subprocess - have
> I overseen something?
BLT has 'bgexec', but maybe it reuses enough of the Tcl API to be
"catchable".
And I wouldn't bet absolutely nobody ever dreamt of calling system()
from an extension...
-Alex
Gerhard Reithofer
On Sat, 28 Jul 2007, Helmut Giese wrote:
> On Sat, 28 Jul 2007 16:29:08 +0200, Gerhard Reithofer
> <gerhard....@tech-edv.co.at> wrote:
>
> Hi Gerald,
Gerhard ;-)
> >I'm aware that Joe's proposal was defined that way, but my problem is
> >that I don't have a feeback what really happens in the spawned process
> >if I do not reread the enviroment, therefore I cannot "control" it.
> AFAIK this won't work because the environment is a "one way street":
> You can set it up any way you want and spawn a child process and this
> process might change it - but you / the parent won't be able to see
> these changes. If you re-read the environment, you will ever only get
> your own.
> At least that's what I learned and I have never seen any different
> setup.
ok, I seemed to be arguing unclear.
I never meant that the change of a childs process will influence the
parents environment, but all environment variables of the current
process will be present in the childs process.
With "controlling" I meant that I request a specific situation and I can
be sure that it really exists in the current and all childs processes.
But if I do a
export VAR=IsSet (in Unix)
or
SET VAR=IsSet (in Windows)
I CAN be SURE that the child process has an environment variable VAR
with the value "IsSet".
If I do a
set ::env(VAR) ""
I CANNOT be SURE that a Variable Var exists with the value "" (meaning
empty string - in Unix it exists in Windows it does not). Nevertheless
if I enter the shell commands
SET TEST=""
or
export TEST=""
in Tcl the variable ::env(TEST) exists with the value "" (meaning two
quotes) on windows and in Unix an empty string is stored.
Similar situation with the case-sensitivity:
set ::env(TEST) val1
set ::env(test) val2
results in a variable TEST with value "val2" under Windows and "val1"
under Unix.
This is the reason why I say that it is necessary to reread the
env-array to be sure which value WILL be present in the CURRENT and all
SUBPROCESSES and even if there will be a any variable.
To close this discussion.
I never wanted to change the environment handling of the underlying
sytem, but IMHO the Tcl-syntactic treatment of the ::env "array" (and
the [info ...] behaviour) makes it hard to work with it as transparent
and platform independent method.
@Alexandre: there are specific windows-only functions in the TWAPI (e.g.
resolving the symbolic Class IDs - like CSIDL_APPDATA), but I haven't
found anything regarding the (historical based) basic env handling.