handing of undefined tcsh variables on context files

[Fede Naum]

Hi,

As you might know tcsh and bash handle environment variables in a different way.

The most annoying thing on tcsh is that if you are setting a variable with a  value that contains a variable , it tries to evaluate it and if the variable is not set or does not exists it dies with the Undefined variable error

for example (provided LD_LIBRARY_PATH was note previously set

TCSH
> setenv LD_LIBRARY_PATH '/my/path':${LD_LIBRARY_PATH}
LD_LIBRARY_PATH: Undefined variable.

while bash just handles that nicely and if the values does not exists it just set that to an empty string.

BASH
> export LD_LIBRARY_PATH='/my/path':${LD_LIBRARY_PATH}
> echo $LD_LIBRARY_PATH
/my/path:

The current implementation of the rez tcsh/csh plugin does not handle this case, hence when you are doing a rez-env from a tcsh where there is one of this such case, it just dies and it is unable to create the environment.
Here at AL were are migrating to bash but we might need to support tcsh for a while.

I tried the new command definition on REZ 2.0

commands: |
  LD_LIBRARY_PATH.append('${LD_LIBRARY_PATH}')

and it ends up with
Error in rex code: NameError - name 'LD_LIBRARY_PATH' is not defined


I just did a crude implementation to handle this case on the tcsh plugin which ends up creating the following lines on the context file (/tmp/rez_XWZ/context.csh)

if ( $?LD_LIBRARY_PATH) then
   setenv LD_LIBRARY_PATH "/my/path:${LD_LIBRARY_PATH}"
 else
   setenv LD_LIBRARY_PATH "/my/path"
endif

That solves the easy example case, but as I can not evaluate anything at the time that I'm building the context file (because I was told that the baked context file can then be used in a different environment)  the implementation to solve more complex cases gets trickier.

So, I just wanted to know if you were aware of the issue and open the discussion if it was not already opened.

Cheers
Fede

[Chad Dombrova]

commands: |
LD_LIBRARY_PATH.append(‘${LD_LIBRARY_PATH}’)

that should be:

commands: |
  env.LD_LIBRARY_PATH.append('${LD_LIBRARY_PATH}')

​

I'm not sure that that completely fixes your problem, but it should resolve your NameError exception with 2.0. 

chad.

[allan.johns]

Hi Fede,

Chad is right that should fix the problem. If it doesn't then it's a bug, there's code to deal with this case specifically. Let me know if so and I'll look into it.

Thanks,
A

[Fede Naum]

Thanks Chad, Allan:

It does fix the NameError exception, but does not solve the Undefined variable issue

it ends up writing

setenv LD_LIBRARY_PATH "${LD_LIBRARY_PATH}"

so in tcsh AFAIK there is no way around it, you have to if/else that line in case the variable did not exists.

Fede

PS: I just did a git pull to get the latest and after workaround a couple of things I tried again with no luck, I still get the same error.

FYI. The 2 things that I had to workaround might be due to my local install, but just in case here they are:

1) I need to pass --no-bootstrap  to rez-env
     - since if not set is  _get_rez_dist_path() or rez utils returns None if the path does not exists hence the validate on the setting module complains.

     - I think that just returning an empty string might be better idea

2) I don't know how to specify a version on the package yaml so it does end up with a Version type.

     - so I changed line 591 on resources.py from
           
       'version': Version('1.2'),
           to
        'version': OneOf(lambda: None, 'str', 1.2),

--
You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/BiOYf7ZvT0A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[allan.johns]

On Monday, June 2, 2014 6:38:43 PM UTC-7, Fede Naum wrote:

Thanks Chad, Allan:

It does fix the NameError exception, but does not solve the Undefined variable issue

it ends up writing

setenv LD_LIBRARY_PATH "${LD_LIBRARY_PATH}"

so in tcsh AFAIK there is no way around it, you have to if/else that line in case the variable did not exists.

Ok I know what this is... there is a fix for this but it's only used in certain cases right now. I'll fix this soon so it applies always.
 

Fede

PS: I just did a git pull to get the latest and after workaround a couple of things I tried again with no luck, I still get the same error.

FYI. The 2 things that I had to workaround might be due to my local install, but just in case here they are:

1) I need to pass --no-bootstrap  to rez-env
     - since if not set is  _get_rez_dist_path() or rez utils returns None if the path does not exists hence the validate on the setting module complains.

     - I think that just returning an empty string might be better idea

If this isn't a showstopper then I'd just hold tight, we're in the middle of reworking the installation process and the way bootstrapped packages are managed.
 

2) I don't know how to specify a version on the package yaml so it does end up with a Version type.

     - so I changed line 591 on resources.py from
           
       'version': Version('1.2'),
           to
        'version': OneOf(lambda: None, 'str', 1.2),

I'm sorry I don't follow, what's the problem you're trying to fix here?

thx
A
 

On Tue, Jun 3, 2014 at 2:17 AM, allan.johns <nerd...@gmail.com> wrote:

Hi Fede,

Chad is right that should fix the problem. If it doesn't then it's a bug, there's code to deal with this case specifically. Let me know if so and I'll look into it.

Thanks,
A

On Monday, June 2, 2014 7:52:58 AM UTC-7, Chad Dombrova wrote:

commands: |
LD_LIBRARY_PATH.append(‘${LD_LIBRARY_PATH}’)

that should be:

commands: |
  env.LD_LIBRARY_PATH.append('${LD_LIBRARY_PATH}')

​

I'm not sure that that completely fixes your problem, but it should resolve your NameError exception with 2.0. 

chad.

--
You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/BiOYf7ZvT0A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rez-config+unsubscribe@googlegroups.com.

[Chad Dombrova]

one thing that is unclear to me is why are you appending LD_LIBRARY_PATH to itself? Is that really the behavior you’re after?

if you do this:

env.LD_LIBRARY_PATH.append('$BAR')

it should work as long as BAR exists, even if LD_LIBRARY_PATH did not previously exist. however, if BAR does not exist, I think it’s an error, which is valid behavior. Also, I believe this should work:

if env.BAR:
    env.LD_LIBRARY_PATH.append('$BAR')

that module has changed around a lot, so I can’t remember the exact syntax. the point is you should be able to check the value of BAR.

chad.

​

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[allan.johns]

Actually that's right, Chad has reminded me why I hadn't applied the feature I'd mentioned across the board. On the one hand I can see that you'd want the commands to just 'work' when referencing an undefined variable, but on the other hand this deviates from tcsh behaviour.

The case where I stop the undefined variable error from happening is where a variable has been configured specifically to be able to inherit from the parent environment. This can be done with the "parent_variables" setting in rezconfig - but it's there mostly to help with migration efforts for studios moving to Rez, in general you wouldn't use this (see the comments in rezconfig for more info).

Chad I think I recall your example code not working, ie:

if env.BAR:
    env.LD_LIBRARY_PATH.append('$BAR')

I can't remember though, I have to revisit Rex again. However, this definitely works:

if defined('BAR'):
    env.LD_LIBRARY_PATH.append('$BAR')

...although I think it should be renamed 'envdefined' or 'env_defined', or perhaps 'env.defined'.

A

To unsubscribe from this group and all its topics, send an email to rez-config+unsubscribe@googlegroups.com.

To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rez-config" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.

[Fede Naum]

On Tue, Jun 3, 2014 at 11:49 AM, allan.johns <nerd...@gmail.com> wrote:

On Monday, June 2, 2014 6:38:43 PM UTC-7, Fede Naum wrote:

Thanks Chad, Allan:

It does fix the NameError exception, but does not solve the Undefined variable issue

it ends up writing

setenv LD_LIBRARY_PATH "${LD_LIBRARY_PATH}"

so in tcsh AFAIK there is no way around it, you have to if/else that line in case the variable did not exists.

Ok I know what this is... there is a fix for this but it's only used in certain cases right now. I'll fix this soon so it applies always.

are you talking about calling the self.interpreter._saferefenv(key) for every variable that contains a variable on the value?

anyways,... no rush, we'll wait for it, we are ok for the moment with a workwaround that I added to the csh plugin, and I'll remove it when that gets fixed.

 

 

Fede

PS: I just did a git pull to get the latest and after workaround a couple of things I tried again with no luck, I still get the same error.

FYI. The 2 things that I had to workaround might be due to my local install, but just in case here they are:

1) I need to pass --no-bootstrap  to rez-env
     - since if not set is  _get_rez_dist_path() or rez utils returns None if the path does not exists hence the validate on the setting module complains.

     - I think that just returning an empty string might be better idea

If this isn't a showstopper then I'd just hold tight, we're in the middle of reworking the installation process and the way bootstrapped packages are managed.

no, is not a  showstopper. We are using an earlier version of the 2.0 branch, I just did a pull in order to see if the latest changes might solve my problem and found that little issue.

 

2) I don't know how to specify a version on the package yaml so it does end up with a Version type.

     - so I changed line 591 on resources.py from
           
       'version': Version('1.2'),
           to
        'version': OneOf(lambda: None, 'str', 1.2),

I'm sorry I don't follow, what's the problem you're trying to fix here?

After I pulled the latest code from the branch 2.0, I found that when I was doing a rez-env it failed. The error was something like it was expecting the version specified in the package.yaml to be of the type/class Version, but our packages.yaml contains a string like 6.5 or 1.0.0. There is a bit of code somewhere that takes the values from the package.yaml and tries to guess the types and then validates them against the reference of an schema (in this case the VersionPackageSchema_0) .

i.e In my case I had 2 packages

name: CentOS

version: 6.5

and another

name: foo

version 1.0.0

the code would guess the type of 6.5 as a float and 1.0.0 as a string, that's why I change that line for 

        'version': OneOf(lambda: None, 'str', 1.2), 

but as I said it might be that the way to specify the version on the package.yaml has changed (even tough I was told that the package.yaml are backwards compatible) 

 

To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

[Fede Naum]

Thanks Chad and Allan,

I had a read at the parent_variables setting in rezconfig, and also looked at the code, but I'm don't think this is going to solve the issue. (not for tcsh). if you leave unexpanded values then at the source time if any does not exists tcsh will keep complaining. 

Also having in mind something that I discussed with Mark, I think that having something like

if env.BAR: env.LD_LIBRARY_PATH.append('$BAR')

won't work either, because this is just checking that the variable is set at the time that the context.csh file is created. Something that I understood from my discussion with Mark is that you might want to use a generated context file at a later time and potentially in a different parent environment where the variable might/might not be set.  

On the other hand if my understanding is wrong then there should be no need to do the if env.BAR on the package.yaml since that can be handled more easily in the shell plugins by just doing a 

if 'BAR' in os.environ 

   write a  setenv/or equivalent for that particular shell

else

   you set the value but remove the ${BAR} bit

what is the behaviour of BASH

 

I think we have more complicated cases that just appending to itself, but currently any simple undefined variable will throw the whole thing down.

I'm not 100% sure that the case we discussed with Mark was a contrived example, so I'll discuss it again with tomorrow and let you know.

Thanks again

Fede

To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.

[allan.johns]

On Tuesday, June 3, 2014 6:57:45 AM UTC-7, Fede Naum wrote:

Thanks Chad and Allan,

I had a read at the parent_variables setting in rezconfig, and also looked at the code, but I'm don't think this is going to solve the issue. (not for tcsh). if you leave unexpanded values then at the source time if any does not exists tcsh will keep complaining. 

You're right in this case it doesn't help (but it's not meant for this case either, so that's ok).
 

Also having in mind something that I discussed with Mark, I think that having something like

if env.BAR: env.LD_LIBRARY_PATH.append('$BAR')

won't work either, because this is just checking that the variable is set at the time that the context.csh file is created. Something that I understood from my discussion with Mark is that you might want to use a generated context file at a later time and potentially in a different parent environment where the variable might/might not be set.  

In rez-2, contexts are stored into .rxt files, which are *pre*-interpreted. Interpretation happens when that context file is consumed, either via the new API or using the cli command "rez-env --input". So the above will work - control flow is delayed until interpretation time. This is different to Rez-1, where interpretation happened at the same time the context was created, but this meant (A) no control flow capabilities and (B) bash centricity.

 

On the other hand if my understanding is wrong then there should be no need to do the if env.BAR on the package.yaml since that can be handled more easily in the shell plugins by just doing a 

if 'BAR' in os.environ 

   write a  setenv/or equivalent for that particular shell

else

   you set the value but remove the ${BAR} bit

It is possible to make this work, the question is whether we should, because it deviates from tcsh behaviour. If we did add this then I would say it should be a configurable option in the tcsh shell plugin, which defaults to False. I'd still like to find out more about your particular use case though.
 

what is the behaviour of BASH

Bash expands undefined variable references to the empty string.
 

[Fede Naum]

In rez-2, contexts are stored into .rxt files, which are *pre*-interpreted. Interpretation happens when that context file is consumed, either via the new API or using the cli command "rez-env --input". So the above will work - control flow is delayed until interpretation time. This is different to Rez-1, where interpretation happened at the same time the context was created, but this meant (A) no control flow capabilities and (B) bash centricity.

That's sound like would solve the problem then.
just to double check, does the consumer process contemplates the case where the variable is not in the parent environment, but it is defined in a package that get's included earlier

i.e
the variable MOUNT_POINT is not defined in the parent environment,

CentOS and Windows package define MOUNT_POINT

CentOS    does not define a MOUNT_POINT
Windows  MOUNT_POINT=C:/

Then a package X uses that like

env.MY_PRESET.setenv('/film/presets/')

if env.MOUNT_POINT: env.MY_PRESET.prepend('$MOUNT_POINT')

So, if I pre-bake this context in linux, would I then be able to consume the .rtx file on windows and get the MOUNT_POINT appened?

 

On the other hand if my understanding is wrong then there should be no need to do the if env.BAR on the package.yaml since that can be handled more easily in the shell plugins by just doing a 

if 'BAR' in os.environ 

   write a  setenv/or equivalent for that particular shell

else

   you set the value but remove the ${BAR} bit

It is possible to make this work, the question is whether we should, because it deviates from tcsh behaviour. If we did add this then I would say it should be a configurable option in the tcsh shell plugin, which defaults to False. I'd still like to find out more about your particular use case though.
 

Well I'm not sure if deviating from tcsh behavior is bad in this case, I mean, at the end of the day what you want is to write a package yaml that ends up resolving an environment with the same value for the variables regardless which shell you are using, right?

An  option in tcsh plugin to manage this behavior makes sense to me.

Thanks for you time
Fede

[allan.johns]

On Tuesday, June 3, 2014 8:03:22 PM UTC-7, Fede Naum wrote:

In rez-2, contexts are stored into .rxt files, which are *pre*-interpreted. Interpretation happens when that context file is consumed, either via the new API or using the cli command "rez-env --input". So the above will work - control flow is delayed until interpretation time. This is different to Rez-1, where interpretation happened at the same time the context was created, but this meant (A) no control flow capabilities and (B) bash centricity.

That's sound like would solve the problem then.
just to double check, does the consumer process contemplates the case where the variable is not in the parent environment, but it is defined in a package that get's included earlier

Yes, a variable set by an earlier package is seen by a later package. This is necessary because some packages depend on variables set by their dependencies.

What is meant by an 'earlier' or 'later' package needs some explanation. If A requires B, then B's commands will be sourced before A's, because A might depend on them. So even if you request "A B" in that order, the actual order will be "B A". However, wherever possible, Rez will keep the order of packages that were requested.
 

i.e
the variable MOUNT_POINT is not defined in the parent environment,

CentOS and Windows package define MOUNT_POINT

CentOS    does not define a MOUNT_POINT
Windows  MOUNT_POINT=C:/

Then a package X uses that like

env.MY_PRESET.setenv('/film/presets/')

if env.MOUNT_POINT: env.MY_PRESET.prepend('$MOUNT_POINT')

So, if I pre-bake this context in linux, would I then be able to consume the .rtx file on windows and get the MOUNT_POINT appened?

Yes, kind of. Two points:

1) This isn't a good example, because more than likely the one rxt file won't be valid on two different platforms. Rez represents platforms as packages, so a resolve on Linux would be different from the same resolve on Windows - different packages might have different variants for different platforms and so on. An rxt file is a *bake*, it doesn't make sense to apply that bake on different systems. That's not to say that we shouldn't allow for somehow regenerating the same context on a different platform (or approximately the same) but that's another conversation.

2) That aside, the answer to your question is yes, although your specific example won't work. This will fail:

if env.MOUNT_POINT:


..when MOUNT_POINT is not defined (you'll get a RexUndefinedVariableError). Here is how you do it:

if defined('MOUNT_POINT'):

You *should* be able to do this, but it's broken (although I've just fixed it in a different branch):

if 'MOUNT_POINT' in env:

Every function used within env has an analogous free function. To illustrate, these rex code snippets do the same thing:

if 'FOO' in env:
    env.BAH = 'hey'
    env.EEK.append('dude')
    env.FEE.prepend('fa')

--vs--

if defined('FOO'):
    setenv('BAH', 'hey')
    appendenv('EEK', 'dude')
    prependenv('FEE', 'fa')

Please used "defined" rather than "if 'X' in env" for now, until the fix is merged with 2.0.
 

 

On the other hand if my understanding is wrong then there should be no need to do the if env.BAR on the package.yaml since that can be handled more easily in the shell plugins by just doing a 

if 'BAR' in os.environ 

   write a  setenv/or equivalent for that particular shell

else

   you set the value but remove the ${BAR} bit

It is possible to make this work, the question is whether we should, because it deviates from tcsh behaviour. If we did add this then I would say it should be a configurable option in the tcsh shell plugin, which defaults to False. I'd still like to find out more about your particular use case though.
 

Well I'm not sure if deviating from tcsh behavior is bad in this case, I mean, at the end of the day what you want is to write a package yaml that ends up resolving an environment with the same value for the variables regardless which shell you are using, right?

An  option in tcsh plugin to manage this behavior makes sense to me.

I can see your point, I'll look into this soon.
 

[Fede Naum]

Hi Allan,

Thank you so much for your clarifications

We are pretty happy with Rez ;)

I really appreciate your time and effort,

Fede

--
You received this message because you are subscribed to a topic in the Google Groups "rez-config" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rez-config/BiOYf7ZvT0A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rez-config+...@googlegroups.com.