Environment variable length limits on windows

[Thorsten Kaufmann]

Hi there,

we expect this to become a problem here rather sooner than later. Anyone had any experience with that?

I took a look at the current shell plugin for cmd and it uses set to apply the environment variables. set does have a limit that is not present when setting the var otherwise (e.g. in python using os.environ[].

I'd be curious if anyone looked into this at some point?

Cheers,

Thorsten

[yawpitch]

I spent a bit of time looking into this, and it's quite complicated.

In Windows versions < Vista the ENTIRE environment block could be a maximum of 32,767 bytes long (so 32,767 ASCII characters, less with Unicode obviously).  So that meant the sum total of all environment variables AND their values couldn't take up more than that amount of space. Technically that meant that the length of any one env var was limited only by the number of other env vars and the length of both their keys and values, however some variables (especially PATH) would cease to function correctly at > 2048 bytes in length.

In Windows versions > Vista (I'm going to hope that anyone working today isn't still on anything that archaic) it's 32,767 bytes per environment variable ... however with the BIG caveat that many programs simply won't know how to work with anything anywhere near that long.  For instance the PATH variable will begin to show some strange behavior (depending on how the process interacts with it) at > 2048 bytes, and get really strange at > 4096.

In all cases (and all Windows versions) the SET and SETX commands are themselves limited by the maximum command line entry length, which is usually 1024 bytes.  Python and other programming languages can easily bypass the command line limit, but shell scripts and especially direct command line calls (ie via Python's subprocess.Popen) might still be arbitrarily limited.  Obviously this can also cause a huge problem when, for instance, a command line call is made with the %VAR% syntax and expansion results in a command line that is greater than 1024 bytes post-expansion.

Interestingly with subprocess.Popen it's fine to pass > 1024 bytes in the environment dictionary itself, you just cannot pass it to the "cmd" variable itself that ultimately ends up being used to create the subprocess, at least with shell=True (the only way I tested it).

Yes, it could become a problem, in other words, and especially if Rez is ultimately passing back "set" or "setx" commands (or running same via subprocess).

M

[Thorsten Kaufmann]

Interesting points and i cam to mostly the same conclusion. Due to the limit beyond the control of rez (like set and setx as well as READING env variables) this can be quite a harsh limit.

I guess it makes sense to think about alternative ways especially for %PATH%. Also it should at least sanely warn you or fail the resolve if one hits these limits.

Cheers,

Thorsten

[Allan Johns]

Windows, who invited that guy?

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Thorsten Kaufmann]

https://media.giphy.com/media/26tPlN7OSKiuUDp6w/giphy.gif

[b.f...@toonboxent.com]

Actually windows is not better or worse than Linux.
It is not the environment variables though.

What we have hit is that the shebang #! size is limited to 127 characters in Linux.
This limit is compiled into the kernel.

Why does this matter?
If you install python tools with setup.py that happen to have a launcher, where your python package lives within a rez space somewhere deep down on the network, with variants, os, arch etc. you might end up with invalid shebang to this python executable.

Easy fix:
Rebuild the kernel for all of your studio (not) or hack the build process of python packages or be lucky to just have started with rez and simply move and rebuild rez packages to live in a path closer to /

Cheers,
Blazej

[b.f...@toonboxent.com]

As to set and setx: At least there is an easy alternative.
On windows you can change the env in the registry and send a winapi command to refresh the environment (for all new processes that is).

I do not own the code any more but I guess this can be easily googled.

Cheers,
Blazej

[yawpitch]

Ahh the registry ... the entire reason why "actually windows is not better or worse than Linux" is so horribly, horribly wrong.

[Thorsten Kaufmann]

Hey there and thanks for the input everyone. I am aware that the limit is higher when using python to set and access the variables. I am hesitant to do that though as it would result in a shell that is not fully functional as any tool that uses standard shell commands to access the variables would break. It's just not a fully functional shell. It may make sense to add this for special cases (like python packages appending to PYTHONPATH for pure python consumption.).

Cheers,

Thorsten

[b.f...@toonboxent.com]

Please read again. It sounds not like something I said in this context and I certainly don't feel comfortable being held advocatus diaboli for windows :)

As to the environment issues:
Are you sure that access is limited? From my understanding only if applications tried to access the environment variable through something like a batch script it would create issues. I can imagine that Python would be perfectly fine accessing the environment as they use something like the winapi. A look into the Python source might help.

In worst case rez could append environment variables instead of full paths.
The ROOTS are guaranteed to be in the environment. I wonder how to tell rez to not expand them, though.

Cheers,
Blazej

[Thorsten Kaufmann]

Well python can access it fine for sure. But as i do not know beforehand what tools will be part of the configured shell there may be tools that can not. If all else fails it should still give a warning i guess.

I just took another look at cmd shell implementation and realised it uses "cmd /Q /K" to execute commands that are written to a file before. Is there any reason it needs to be done like that? Can't i just use subprocess.call() on the lines of the list rather than execute it using cmd.exe?

Cheers,

Thorsten

[Thorsten Kaufmann]

I'll take that back as that actually spawns the shell. Need to look into that some more and why it can't call doskey etc.

[jri...@rodeofx.com]

Hello Thorsten,

Here at Rodeo, we are currently looking into Rez. We are a studio supporting all the 3 OS, mac, Linux and Windows.

We have the same concernes about the environment on windows.

Did you have time to test with rather long environment variables for PATH or PYTHONPATH?

Best,

Jordi

[Thorsten Kaufmann]

We did not do any real testing, but i spend a while thinking about this. I'd say it depends a lot on different factors, mainly two seem relevant to me:

- How you define and structure your packages

- How you scope your environments/contexts

- How variables are actually set

- Where your packages are located and how you access them

So in regards to the first part, there are different possibilities. You can just blindly append to PATH for every packages binary folder, you could try exposing just executables as seperate env vars (e.g. REZ_ENV_TOOLNAME_EXECUTABLE) instead of appending path etc etc. That is very dependent on the package type and in some cases the particular package. But there are also other factors playing a rather big role here. For example variants. Take python packages that come in variants for python version, architecture, os and VS version on windows (the latter being needed if you have binary extensions that need special compiles for nuke or maya).

This may result in the path added to PATH growing from this:

\\server\packages\package_name\1.0.0\

to this:

\\server\packages\package_name\1.0.0\python-2.7\platform-windows\arch-AMD64\os-windows-6.1.7601.SP1\msvc-9.0

The second is quite a big factor. What i mean by scope is "How many packages are in your environments". From what i read there seem to be quite some shops that have big environments. Say a single environment for a show. That means there would be a LOT of packages in there and the PATH variable (and PYTHON_PATH and possibly others) may

grow rather uncontrollably if you take dependencies into the mix etc. We are rather planning to have different contexts for different contexts (hah, contextception). So for example there would be separate contexts available for different tasks, rather than few monolithic contexts.This way the package count is way lower within environments/contexts.

The third part is something that we should definitely take a closer look at. Especially since some of the approaches currently taken also have side effects. PATH is currently taken from the registry for cmd, which means you can not override in an existing shell as local PATH changes are not written to the registry of course. For this issue though the main reason would be that you have different amounts of characters available depending on how you set the variables. I have not really spend any time on this yet though.

Our rez install is on a central server, we access it through a local symlink though. That can reduce the amount of characters per path quite a lot, also the path depth of your packages folder itself matters of course etc.

All in all, it is hard to make a general recommendation i guess. It is an issue though and some parts of rez (or rather the shell plugin for cmd) definitely need a closer look.

Cheers,

Thorsten

[Blazej Floch]

Just to add some thoughts and numbers.

We're mostly Linux based, our maya environment there (plugins, scripts as package, lot of in-house dev and lib dependencies) has PYTHONPATHS / LD_LIBRARY_PATHS => PATHS in the range of 9000 characters. Potentially growing in the future.

If we'd strip out the root path, so basically have the rez packages live in something like /rez/ or R:\ it would still be around 5000 characters.

One thing is certain: CMD will not work. And it is unlikely that microsoft will spend time fixing this.

However it is worth researching if you can circumvent the limits by using the Windows API directly to set process environment and spawn an environment. Maybe there is even replacements of cmd we could use instead.

I still found some people claiming that there is a 1920 character limit for PATH. I wonder if this still applies as the 32k for a shared environment limits is said to be gone in modern windows systems. I am not so up to date into ways to deal with dll discovery in windows these days. Maybe there is an alternative model that would work better (but still be dynamic/controllable per process).

We will not have the capacity to do the research as windows is not our main production environment. But I'd love to hear if someone will find a way (if there is a way at all).

Cheers,
Blazej

[Thorsten Kaufmann]

IT here did a bit of testing a while back (thanks Christian if you read this!). Testing was done from python in a standard cmd shell. And it may not be possible to get around cmd in all cases. Anyways the main culprit is accessing long variables in CMD itself (or in a batch file for that matter). If it is longer than approx 8k characters it is not shown at all anymore, when appending it is probably truncated. Another limit: In the registry you can only save a max length of 16383, but a native application even when launched through CMD still can see 32k just fine it seems.

- Creating a variable from python with length 32734 works fine in python itself

- Accessing the variable in a shell (using echo) fails

- Accessing the variable in powershell (using $env) works

- Accessing the variable using the win32 api (using ctyps) works

- Accessing the variable from python as launched via os.system and cmd exe also works

I proposed thinking about a powershell plugin before, and it may be worth a look. In general i personally am not using the generated batch files and alike and i am wondering if anyone on windows needs those or if these could be dropped altogether because they

seem to be an extra hindrance in moving this to the last century ;)

Cheers,

Thorsten