Custom setup steps during env creation that needs information about other packages

[Thorsten Kaufmann]

Sorry for the vague title, had trouble finding a more concise one.

Our main 3D DCC is 3dsmax, so i am looking into getting this to work with plugins as it is a hard requirement for us.

Here is the scenario:

There will be multiple 3dsmax packages for different versions. These will be just pointing at local installs.

There will be a host of different plugins as versioned packages with variants for the different Max versions

To construct the environment the following will have to happen:

• set up all the plugin packages' environments. This  may involve the following:
  
• Set and expand environment variables (no problems there)
• Eventually copying files locally (can this be done?)
• set up the max environment (now the tricky part)
• set up the basic max environment (env vars and what not, all green)
• dynamically create an ini file that includes all of the plugins that were requested for the env
• launching max supplying that very ini file

In our current setup we have .ini files for each plugin and include them in a combined ini file. This needs to become dynamic. I can either generate a dynamic plugin.ini that simply points to the packages

or have individual plugin.inis (that need to be dynamically generated too as they can not have relative paths) that then need to be dynamically included in the main plugin.ini.

So this leaves me with the following questions for now:

1. Is there a best practice to share information like "i am a max plugin" categorisation with the host package? Or any other information for that matter?
2. What is possible in package.py? As far as i understood every line must be self-contained. But can i have something like say an "ini file generator module" that i supply with info from the resolve to create the ini in one line? Also what about manipulating env variables beyond appending/prepending/setting. I'd like to do things like "add if not present" or "only set if it does not exist" (don't override)
3. What is the status of rezbuild? There was mentioning that i should resort to CMake wherever possible. I don't really know CMake though and the rezbuild.py feels comfortable. Especially since i hardly build anything yet but merly copy things around etc.

Cheers and thanks in advance,

Thorsten

[Sebastian Kral]

Hi Thorsten,

we built such a launcher which prepares everything, starts 3dsmax and cleans up after 3dsmax has closed. It's not perfect but does a pretty good job. We are not using rez in production yet but are thinking of how we can combine these to systems. I think it's better to build a separate launcher and not try to extend rez to do this. Rather let rez add packages to a specific environment variable which the launcher recognizes. The launcher is a package itself and can be easily versioned and updated. It can create an alias for the launcher which would be your way of starting max with the launcher.

3dsmax writes a lot of files to disk just for configuration (usermacros, etc.) which has to be cleaned up before you start it with a new configuration. This can be quite tiresome but needs to be done. I don't know rez very well yet but I don't think there is any mechanism which can be invoked after a program has exited in an environment. You might say that at the moment you are in a rez environment the 3dsmax should be configured and ready to start and I agree but how would you handle the clean up reliably?

I am happy to discussing these things off list because I don't think rez can deliver what you are looking for.

Let me know what you think.

Cheers,

Sebastian

[Allan Johns]

Hey Thorsten, Sebastian,

The launcher route that Sebastian describes is what I often do, and it works fine.

You can actually have some program run at environment creation time also though. Assuming your package has, say, '.max-setup' tool available, you can go:

# in package.py

def commands():

    command(".max-setup")

The 'command' function will run the given command when the shell is created.

Sebastian is right that there's no equivalent for shell exit time, however this need has come up for us also recently, and I think it would be straightforward to add support. The idea is that a package would also be able to provide an exit_commands() function, whose interpreted code (remember that it gets converted to bash/etc) would be run at shell exit. I believe that would give you everything you need right?

Thx

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Sebastian Kral]

@Allan I think that would be great. Of course it has to be reliable but if it is, it's a great way of doing this.

To keep people from starting multiple 3dsmaxs in different environments at the same time a lock file should work just fine. We haven't implemented this ourselves yet though.

[Thorsten Kaufmann]

Thanks a lot for the input again. I kind of need to push this into rez rather than having the logic itself in a launcher as i am facing VERY different environments.

I think the main culprit here is that the plugin packages do not know anything about each other.

I think i need to look at the way envs are actually resolved and created. I assume they are first fully resolved and then created, right? So could i inject information into the resolve during resolve per package so that the max package knows which plugins it needs to include in its ini files? And have the plugins provide their ini files?

Cheers,

Thorsten

[Sebastian Kral]

plugin packages do not know anything about each other.

Can you give an example on what they need to know about each other? What more do you need than one package requiring another package?

So could i inject information into the resolve during resolve per package so that the max package knows which plugins it needs to include in its ini files? 

I think you have to decide who plays the active and who the passive role in your setup. It sounds like you want the plugin packages to be passive and present itself to the active max package. In my opinion this cries for a max launcher which would would check an environment variable for a list of plugin paths and prepare max before you start it. I think rez usually works the other way around (plugin active, max passive). Every package is actively changing the environment and the max package would only do what is necessary for itself. (@Allan Please correct me if I am wrong). This would mean every package adds itself to the main plugin.ini which max uses to start. I don't think this is the way to go though I am just trying to think about all the possibilities.

We are creating temporary plugin.ini and 3dsmax.ini files on the fly and start max with them. This way we can just throw them away when we are done and don't contaminate the original ones. This could be done with rez too by extending rez to have something similar to the env object in commands() for the plugin ini and the 3dsmax ini. This way every plugin can add itself to these and they will be written to a file when rez has resolved the environment. The only issue would be how to get max to use these two files. Is there a way to give default arguments to an executable with rez? Probably with an alias or something similar. This would make it possible to set up the max package to always use a custom plugin.ini and custom 3dsmax.ini but instead of hardcoding anything into the package we can use environment variables which point to these files and the default values are the original ones. If we don't overwrite the environment variable with rez it would work as a normal max.

Hmm. Reading my own text shows me that this would include a lot of max specific changes to rez, maybe a standalone max launcher package which adds to the rez functionality would be better?

And have the plugins provide their ini files?

We don't create plugin.ini files for every plugin anymore. We just add the important paths to a custom plugin.ini which we create for every max start. Since a lot of the plugins need so many more modifications to make them work, we thought there isn't much point to it.

Cheers,

Sebastian

[Allan Johns]

@Sebastian, correct. Typically (in Linux at least) a plugin would add itself to the app's PLUGIN_PATH or similar.

Every package has access to the 'resolve' object, so why not just have the max package find any present max plugins, and construct its ini in a tempdir? Save this path to an env-var, say, _MAX_TMPDIR or something, and then have a small wrapper for the max binary, which uses this env-var to points max at these ini files?

The only downside is that there's no opportunity to clean up the tempdir. There are 2 options I think, both are new rez features but not big ones:

1) Rez already has its own tempdir; it would be trivial to create a subdir here that's for package use. Packages could put whatever they want there, and rez would clean up on exit (as it already does);

2) We add the exit_commands() package functionality. These commands would run at shell exit time. In this model, max would create its own tempdir, write the path to an env-var, then use that env-var to cleanup the dir on exit.

A

[Thorsten Kaufmann]

Hey there,

good points. I was not aware that i can save arbitrary information to the resolve object. I think that makes most sense. I had thought aboute using env variables but i think i prefer the resolve object (for no particular reason whatsoever actually heh)

Assembling a full ini in max of course also works. The only difference is that i am adding paths instead of ini files in the include section. The more i can put into rez (without hacking it) the better from my current point of view. But i will need to do the plugin.ini generation in a max wrapper unless i can force max to be evaluated last because if max is evaluated before a plugin it will miss it. Unless of course the resolve object is created during resolving before actually creating the environments, then i could stick it into the resolve object and it should have all.

In regards to cleaning up, we would need that anyways. But i think that for different purposes both features would make sense. e.g. the plugin.ini in the package temp vs. other setup/teardown stuff that has to happen elsewhere (e.g. making changes to a locally installed application's folder or alike).

Cheers and thanks again!

Thorsten

[Allan Johns]

Hey,

So I need to clarify a few things, see below:

On Thu, Apr 21, 2016 at 9:19 AM, Thorsten Kaufmann <insti...@gmail.com> wrote:

Hey there,

good points. I was not aware that i can save arbitrary information to the resolve object. I think that makes most sense. I had thought aboute using env variables but i think i prefer the resolve object (for no particular reason whatsoever actually heh)

Oh I didn't mean that you can save arbitrary info into the resolve object. Maybe you can, but you shouldn't! I can see the value in setting aside an object for this purpose though, it's an interesting idea.

 

Assembling a full ini in max of course also works. The only difference is that i am adding paths instead of ini files in the include section. The more i can put into rez (without hacking it) the better from my current point of view. But i will need to do the plugin.ini generation in a max wrapper unless i can force max to be evaluated last because if max is evaluated before a plugin it will miss it. Unless of course the resolve object is created during resolving before actually creating the environments, then i could stick it into the resolve object and it should have all.

So two things:

1) Package command evaluation order is controlled by package dependency first, and request order second. So if a max_plugin package requires max (as it should), max will be evaluated *first*. That's because it might set an env-var that the plugin expects (for example, it might set a PLUGIN_PATH that a plugin then appends to).

2) The 'resolve' object contains all packages up front, so even if max is evaluated first, it can at least use 'resolve' to iterate over available packages, and get their name/version. You can't get to arbitrary package attributes this way though.

3) If you need max to evaluate after the plugins (perhaps the plugins are setting env-var(s) that you need to use), you can use the post_commands() function. There are actually 3 functions - pre_commands(), commands(), post_commands(). They all do the same thing and run in the same order, however they are done in 3 phases - pre first, then commands, then post. So, assume that none of your plugins have their own post_commands(), max's post_commands() will run *after* the plugins are evaluated.

 

In regards to cleaning up, we would need that anyways. But i think that for different purposes both features would make sense. e.g. the plugin.ini in the package temp vs. other setup/teardown stuff that has to happen elsewhere (e.g. making changes to a locally installed application's folder or alike).

Cheers and thanks again!

Thorsten

Nw, hth

A

[Thorsten Kaufmann]

Thanks a lot again for the clarification. I'd really like to see the ability to add arbitrary information to resolve sounds like that might be very handy. Using the pre/post commands sounds like it would solve most of our tasks atm.

Also i would really like to see package temp folders!

@Sebastian: In regards to the plugin.ini i will simply generate a random uuid or alike and hand that over in an env variable or alike to be able to launch max in multiple contexts. The goal is to have the Max Program folder untouched and write protected. Though i do know that has some caveats.

Will try and create a minimal working setup today as i think i got most bits and pieces together.

Cheers and thanks again,

Thorsten

[Sebastian Kral]

@Thorsten Can you share how you ended up doing things? Would be interested to see if you got it working with rez.

[Thorsten Kaufmann]

Not at all Sebastian,

so my main use case is 3dsmax + plugins. I'll try and leave unrelated bits out.

For the plugins i have a rather standard (at least i guess it is ;) ) variants based setup and i am creating a new env variable that all plugins append to like this:

    env.MAX_PLUGINS.append(r"{root}\Program Files\Autodesk\3ds Max " + str(env.REZ_3DSMAX_VERSION) + r"\plugins")

    env.MAX_PLUGINS.append(r"{root}\Program Files\Autodesk\3ds Max " + str(env.REZ_3DSMAX_VERSION) + r"\plugins\vrayplugins")

Then in the 3dsmax package i have a wrapper that simply creates a plugin.ini from the env var like this:

plugin_filename = os.path.join(tempfile.gettempdir(), str(os.getpid())+"_plugin.ini")

with open(plugin_filename, "w") as ini:

    ini.write("[Directories]\n")

    ini.write(r"local=C:\Program Files\Autodesk\3ds Max 2016\plugins" + "\n")

    for idx, path in enumerate(os.getenv("MAX_PLUGINS", []).split(";")):

        ini.write(str(idx) + "=" + path + "\n")

A few notes here:

• As you can see i am creating a temporary plugin.ini with the PID as part of the filename to ensure being able to run multiple max instances with different contexts
• Not performing cleanup yet. Bad squishy!
• As this is being done in a simple wrapper that is used to launch max i don't care about doing things in pre_commands or post_commands yet

A few notes on the changes i plan:

• I need some more testing on the limits on env variables. when expanded a single path ends up around 110 characters in some common tests i did. So it's easy to get the variable to get quite long. Not sure if there are limits on windows and what they are (esp. through python. I do know that there is a rather low limit when using SET for example. No idea what parts of rez are affected how in this case)
• Depending on the first point, but most likely anyways i may move this to the plugins writing individual plugin.ini to a common temp folder (created and made available via an env variable in pre_commands) and a post_commands step to create a combined plugin.ini simply including all inis in that folder. Will see.
• Add clean-up, error handling and a better more comprehensive wrapper.

Cheers,

Thorsten

[Thorsten Kaufmann]

Not at all = sure can do. Thought you asked if i mind sharing it, heh.