Trying to make a dict in python from hierarchy

[river bell]

hi,

I have a string , which is basically a list of maya objects in a scene with their full path.Something like this

ch_mojo_Master
ch_mojo_Master|EyeBrow_grp
ch_mojo_Master|EyeBrow_grp|L_eyeblow_geo
ch_mojo_Master|EyeBrow_grp|L_eyeblow_geo|L_eyeblow_geoShape
ch_mojo_Master|EyeBrow_grp|R_eyeblow_geo
ch_mojo_Master|EyeBrow_grp|R_eyeblow_geo|R_eyeblow_geoShape
ch_mojo_Master|Mouth_grp
ch_mojo_Master|Mouth_grp|tongue_geo
ch_mojo_Master|Mouth_grp|tongue_geo|tongue_geoShape
ch_mojo_Master|Mouth_grp|upper_teeth_geo
ch_mojo_Master|Mouth_grp|upper_teeth_geo|upper_teeth_geoShape
ch_mojo_Master|Mouth_grp|lower_teeth_geo
ch_mojo_Master|Mouth_grp|lower_teeth_geo|lower_teeth_geoShape

can i create  a dict by splitting at "|" character. So that i have a nested dictionary.

I then intend to write this dict into the file with fileInfo command.

I want to track changes in object names and hierarchy with this solution.

Thanks

[river bell]

edit:

I almost had a solution here : http://stackoverflow.com/questions/8484943/construct-a-tree-from-list-os-file-paths-python-performance-dependent?noredirect=1&lq=1
But it is using default dict rather than a normal nested dictionary.
And i am not able to convert this nested defaultdict to a normal dict.


Thanks

[Justin Israel]

Couple questions...

Is it a problem to use a defaultdict vs a stdlib dict? The benefit is just simplifying the need for checking if a key exists, so that you can freely access it and have it create a child data structure as needed. But in that example they are using files as leaf items and storing them in a list. I assume you don't want that, so that you can have arbitrarily deep hierarchies, so it would end up with leaf items having empty dictionaries, and dealing only in keys.

What are you access patterns? An arbitrarily nested dictionary is a bit complex. Are you just trying to see if something has changed between two sets of data? Are you trying to do various searches within this data structure?

Justin

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/441a0e3a-b444-4d05-a867-83757e4afa4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alok Gandhi]

@river: Here's a gist that you might found helpful:

https://gist.github.com/hrldcpr/2012250

On Thu, Dec 15, 2016 at 6:19 AM, Justin Israel <justin...@gmail.com> wrote:

Couple questions...

Is it a problem to use a defaultdict vs a stdlib dict? The benefit is just simplifying the need for checking if a key exists, so that you can freely access it and have it create a child data structure as needed. But in that example they are using files as leaf items and storing them in a list. I assume you don't want that, so that you can have arbitrarily deep hierarchies, so it would end up with leaf items having empty dictionaries, and dealing only in keys.

What are you access patterns? An arbitrarily nested dictionary is a bit complex. Are you just trying to see if something has changed between two sets of data? Are you trying to do various searches within this data structure?

Justin

On Wed, Dec 14, 2016 at 11:44 PM river bell <river...@gmail.com> wrote:

edit:

I almost had a solution here : http://stackoverflow.com/questions/8484943/construct-a-tree-from-list-os-file-paths-python-performance-dependent?noredirect=1&lq=1
But it is using default dict rather than a normal nested dictionary.
And i am not able to convert this nested defaultdict to a normal dict.


Thanks

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/441a0e3a-b444-4d05-a867-83757e4afa4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0eqrLE%3DpYsOwzeLzWXuseK9ZJWwtNyOeiFOCdKJguMFA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[river bell]

Hi Alok ,

Thank you for taking interest and helping me.

Yeah you are right I am going to use it to check if something has changed in two data sets.I am also going to store vertex count along side each obj name.

I intend to store this in the scene file with fileInfo command.

I am also going to add custom attrs on these objects in the hierarchy to lock in the original name of the object as a second approach.

So I know that I will have to do searches/iteration within this data structure, when comparing between two sets for name change.

My original problem is regarding geometry cache transfer, which acts up upon change in object name.

Our current workflow is as follows:

there are 2 paths our model takes

model >rig> animation

and in parallel

model>texture>shading+lighting

cache file from animation >shading model which is where the name change issue pops up, and we are manually changing names to overcome this,

I intend to do this process with scripts to make this process seamless.

Any suggestions on what data structure to store this data so that traversing and comparing will be easy.

Any other approach you might have as suggestions are welcome.

Thank you so much

Warm Regards

River Bell

[river bell]

hey Alok
Thanks I didn't see your second post in my mail.
But  I saw it in google groups.Its maybe what I was looking for.
will go through it.

Thank you so much again.

Warm Regards to you.

[Marcus Ottosson]

Second post? I'm also only seeing a first post. Did you happen to also not see Justin's post?

This has happened to me before, some replies to a thread, or sometimes entire threads, get caught by a spam filter (Gmail).

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/4bae1961-a7bf-4ad0-b62e-1324cf9067a9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Neil Roche]

Hi River,

Out of interest why does your shading model have different naming to your animation model?  At our studio when the rig is published for animation an alembic of the geometry from the rig is published too, this is what the LookDev and Lighting artists use for shading and ultimately gets the animated cache imported onto.  This also means that the naming stays consistent throughout animation and lighting so you won't have these problems.

It seems to me you are trying to overcomplicate something which would be solved if the names were consistent or is there something else you need this data for?

Neil

[Marcus Ottosson]

Out of further interest, and further off topic, @Neil, when you say gets the animated cache imported onto how do you mean? It sounds like a Maya Geometry Cache workflow, where only point positions are stored in the cache. I was under the impression that an Alembic contains both mesh and point position and thus couldn’t be “applied” to any mesh. Unless you are.. blendshaping?

​

​

[Alok Gandhi]

Yea I guess, Justin's post was added to mine while replying. I should have trimmed.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/fb219e2d-d906-492d-b73f-0dae0cb80ed4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[river bell]

Hey Marcuss

Sorry I dont see how this happened.I somehow mistook Justins post  for Aloks "first Post"

ended up thanking only Alok, so a big thanks to Justin too.

Thank you all infact

Regards

[Justin Israel]

Hi guys,

Thanks for including me! ;-)

Justin

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/5e2a3ff1-ad38-4499-8681-1a5e20e4d922%40googlegroups.com.

[Neil Roche]

Hi Marcus,

Alembic does contain both but when a mesh is deforming it's transforms are locked or static so it will just store point positions per frame whereas if the transform is animated it will store the transform data per frame and the point position remains static.

So for our workflow we just merge the animated alembic cache onto the static lookdev model and that gives us the cache ready to light in the scene with the correct shaders etc.  It's the same as Cache > Import Alembic > Import Under Current Selection with the parent node(s) of your geometry selected.  Not a blendshape in sight ; )

Neil

[river bell]

Hi Neil,

I am a new entrant in the studio.And trying and spruce up the workflow a bit here.
The name changes happen either by the animators in rare cases adding deformers(which shouldn't happen)
or by the back and forth between the rigging and animation department.Currently we are using maya geo-cache.
Have not worked with Alembic before, but what you said seemed like a rock solid solution.
Could you please elaborate a bit more on this workflow , maybe direct me to a good read on the subject.

Thank you

[Marcus Ottosson]

It’s the same as Cache > Import Alembic > Import Under Current Selection

I had a quick look at this just now, and as I suspected, the inMesh is being replaced by the Alembic. What I hadn’t thought of however is that the node itself remains the same which means shader assignments and custom attributes also remain.

If there’s one thing I miss about the Geometry Cache workflow, it was the ability to maintain a mesh in the scene, and simply replace the cache. With Alembic, I lost that and somehow this workflow you mentioned has gone unnoticed. I had simply accepted the loss, as the gain was greater.

So thank you, that is just excellent!

​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/866abebf-705f-4399-8204-f444dedaea27%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Neil Roche]

Hi River,

I don't have any Alembic links but I'm sure if you google it there a re quite a few tutorials and articles.

Are you referencing rigs?  That will mean animators won't be able to rename geo.  Also try and avoid animators adding deformers, get them to pass the rig back to your rigging team for that.

Alembic is great, it can have it's own set of problems but out of the box it is very powerful.  You need to keep your naming and geometry hierarchies consistent per asset though. In our studio when the rigging team publish a rig a new Alembic cache gets created at the bind pose and given to the LookDev artist so if there are any changes in the geometry or hierarchy the LookDev artist reassigns the shaders.  This means that the animated cache and lookDev geometry always match

Also Alembic can be imported into other software so if you are using Houdini for FX you can import the latest animation cache and work from there.

Hope this helps!

Neil

[river bell]

hey Neil,

First up thanks for the excellent input.
Barring a few doubts we have to surely consider this approach soon.

We do reference the rig for animations
I have found that the name change also happens in the texture dep.

From what i know yet alembic needs model uvs in place so the idea of texturing and rigging and animation running in parallel was not possible and therefore not considered as of yet.
I am in a position to research and introduce a better pipeline so will delve this way for sure too

Thanks

[Neil Roche]

No problem!

You should probably find out why your texture department are renaming models, it's not something that is very commonplace.

Also you don't really need finished UVs to start rigging and animating. Once texture have finished the UVs update your rig with the latest mesh and the animators can update to the final reference rig and cache out the finished animation with the latest UVs.

Also Alembic has an option to write UVs to the cache file or not. 

All the best.

Neil

[Marcus Ottosson]

Also you don’t really need finished UVs to start rigging and animating.

Minor nitpick, but it’s a little too general of a thing to say. I know I often find use in having controllers or even functionality depend on the position on a mesh via UVs; even in the early stages. I treat it as one way of overcoming the problem of topology changes in a rig, given the UVs can remain the same, and usually must due to textures.

ps. Say hi to Will Pryor from me. :)

​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/5ef6da35-7ac7-4367-8690-1676d98e8dba%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Neil Roche]

Hi Marcus,

Yeah, ideally the UVs would be finished but sometimes given tight schedules we'll start a rig with a very early model with no UVs just to give us a template for joint position and then update as soon as the UVs are laid out. 

Will says hello too!  : )

[Alok Gandhi]

@river - Start simple (if you do not have any rudimentary pipeline setup). In designing pipeline (and eventually tooling for it) you need to start light but solid with some fundamental principles incorporated.

Core Tenets

Three things are of fundamental importance - Scalability, Reliability, and Security. Each of these aspects should be addressed.

Scalability - The pipeline should be able to handle 10 shots as easily as it should 2000. Although, from the get go it is not easy to design for future proofing the data quantity but still you should pay some heed to it

Reliability - The data should be reliably stored and backed-up. In the case of database outages, the pipeline should be able to recover itself as quickly as possible. If possible, there should be no central point of failure - go as distributed as possible(git is a wonderful example of such a distributed system). With Cloud Computing now becoming affordable and secure each day, you can take a look at it also.

Security - Protecting the IP is important. User / Tool permissions to access the database should be implemented in the most secure way. This could start by harnessing the power of the os user permission (in the case of folders). Users should not be able to access the data only tools should. If you are using a database, most of them have this inbuilt, look at their relevant APIs.

Tools

Some basic tools need to be designed -

Publishers - For exporting data from DCCs / Departments. Naming conventions and validity checks are the first part, then comes the part of exporting the data to a database.

Builders/ Importers - For importing data into DCCs, basically looking up the data in a database (these can be as simple as folders or as complex as large database spread on huge servers)., retrieving it and then importing it in the scene.

Versioning

It also worth mentioning that versioning system is of uttermost importance here as well, do keep in mind what kind of versioning framework you are going to use. It should allow easy versioning operations like versioning up or down and should be intuitive to the artists. They should be able to easily access and use any old version and also able iterate over versions of their work/tasks in the simplest possible way. Again, look at the design principles used in git for versioning. If whole Linux Kernel can depend on it, so can a pipeline.

Inter Studio Data Flows

While talking of inputs and outputs, do mind that at times most modern pipelines have both 'inter-' and 'intra-' data flows. Meaning, you have intra-departments data flows in your studio, but also there is the case of inter-studio transfers. Collaboration is a growing trend in our industry. Even the biggest studios these days are outsourcing their work to smaller shops. This means you should be able to efficiently handle and transfer back the data coming from and to an outside studio. Most of such transfers employ industry-accepted cutting-edge formats like alembic. Thankfully our industry has encouraged and embraced open source projects more than any other. 

Free Open Source Projects

There is a host of open source projects, big and small, helping you to facilitate such pipelines. Alembic is just one them. Also,  USD from Pixar is gaining popularity since it has gone open source. In short, don't reinvent the wheel, use already available libraries to your advantage. This can save huge time and development costs.

Pipelines are living entities

Pipelines are a complex system and have a life of their own. They evolve over time responding to various production pressures. You cannot think of every possible requirement before it comes to haunt you, but you can surely design loosely coupled components (as generic as possible) so that it can evolve easily. You are never allowed to say no.

The Human Element

Pipelines have two brains (just like us human have left and right brain). One is the free flowing, creative, innovative side and the other is logical, calculating and rational. Both of these, fight with each other all the time. The resulting conflict causes the pipeline to evolve. The human side is the artist, they need fewer buttons on the UI, simplified workflows and be able to focus on their art rather than the technology. Tool design should take care of this. On the other hand, there is the database, the data, the metadata and all the stuff like versioning systems, naming conventions, data access, io operations that need to be designed in the most rational, pragmatic and logical way possible. A good pipeline constantly tries to broker peace between these satisfying the common denominator. And of course to wreck all of this are 'the clients'. They are paying and are the boss. They can and do demand most outrageous things and ask it to be delivered at exactly 2.37am the next morning. Do account for that also.

Finally, 

import this

It would be a disservice to write on the python-inside-maya group, without mentioning the awesomeness of python. Go to your REPL and type `import this`. Read carefully the stuff that your REPL spits out. Called 'Zen of Python', these are like the 'ten commandments' (only better, these are 20) of the religion called python. These apply to any design process. Each one is a gem and most of them pertain to pipeline design. Timeless in nature, these are relevant to any sort of engineering. Think about them and how they apply to your pipeline design. Use them in your pipeline and thank Tim Peters for writing 19 of them.

These are my thought, quickly written and somewhat incohesive, but I hope they help.

Good luck!

- Alok.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/3046793b-0896-45f3-9484-efe8a54ea58a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Alok Gandhi]

@river: Coming back to your original question - 

Any suggestions on what data structure to store this data so that traversing and comparing will be easy.
Any other approach you might have as suggestions are welcome.

I think that tree structures implementing python dictionaries sound a bit complicated for the task. I would suggest the plain text as is (eg. ch_mojo_Master|EyeBrow_grp|R_eyeblow_geo|R_eyeblow_geoShape) as you already have it or some other format that suits you and then use some diff tool to obtain the change through the deltas. Take a look at python standard library diff tool library - difflib. It can do the heavy lifting for you. I cannot personally vouch for it as I have not used it in production.

To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/3046793b-0896-45f3-9484-efe8a54ea58a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

[river bell]

Once texture have finished the UVs update your rig with the latest mesh

Hello Neil,

Are you suggesting that we reference the texture file in rigging , mostly it throws up a lot of problems and have been advised not to reference model in rigging.

The other option is to transfer the skinning from the old model to the new one and do touch-ups....Here I am not sure if other kinds of deformers (blendshapes,cluters and wraps) can also be transfered proceduraly.If ithis is possible then I'm reaching somewhere.

Any hints and possible directions I should be taking ?

Thank you.

[river bell]

Hey Alok,

This kind of input provokes respect.Thank you so very much for your time and effort.

This is kinda more than I expected.Awesome !

It would be a disservice to write on the python-inside-maya group, without mentioning the awesomeness of python.- Good line of thought.

quickly written and somewhat incohesive, but I hope they help. Don't think they are in-cohesive though Rich intention >Much appreciated !!! 

Thank you

--

[Neil Roche]

Hi River,

No definitely don't reference models but the rigs should be referenced into the animation scenes and therefore all the nodes are locked and cannot be renamed.

In terms of other deformers they can all be scripted and procedurally created.

Neil

[Marcus Ottosson]

No definitely don’t reference models

This is turning into an interesting thread. :)

On this topic, I’d like to share that I used to think so too, until I found these advantages.

1. It protects the names (like with rigs)
2. It saves space on disk (models can be large, and are otherwise duplicated in every version of a rig)
3. It reduces the time taken to save (related to 2, but important enough to highlight)

Although this assumes that:

1. References are imported on export/publish (nested references are sometimes confusing)
2. The referenced file doesn’t change (like a fixed version. changes may break previously functional rigs)

In my everyday workday, the assumptions are safe to make and benefits are fantastic, especially (3). It means I save more often and as an end result end up with less lost work on crashes and other hazards. For example, if a rig has a dense mesh of 50-100mb, saving on a network can take 3-5 seconds. With a referenced model, saving takes <1 second. Additionally, space savings it enables greater archiving. 100 versions of 100 mb is 1 additional gb for a single asset. In other cases, a mesh can weigh in at 100-500 mb, that’s when you really feel the savings.

Food for thought!

​

[Marcus Ottosson]

Where did my math go? :)

100 * 100 mb = 10 gb for a single asset. Woh!

​

--

Marcus Ottosson
konstr...@gmail.com