is there a built in way to include metadata in EXR coming from Maya-MentalRay

1,433 views
Skip to first unread message

Asi Sudai

unread,
Dec 20, 2013, 2:52:51 PM12/20/13
to python_in...@googlegroups.com
Hi gang,

I'm looking to embed some metadata into renders coming out of Maya&Mentalray.
Would like to embed custom information like scene, user, assets and even plugin versions.

Is there a build-in way ( vie one of the global nodes, attributes in maya ) that will pass the information into mentalray so it can embed it in the EXRs?

My workaround would be to do add the metadata as a second process, after the render is done, but I hope to avoid that if possible... is it?

Ævar Guðmundsson

unread,
Dec 21, 2013, 5:31:51 PM12/21/13
to python_in...@googlegroups.com

My workaround would be to do add the metadata as a second process, after the render is done, but I hope to avoid that if possible... 

There is a way to do as you describe, but why would you want to avoid a second process?

  I know most people frown upon running it as a second process but in most cases where I've come across it's usually what the second process is being used for rather than any despite towards the notion of a second process in it self.

  Some bad ones:
--[ The second process involves starting up secondary application or software.
--[ The second process breaks a lot.
--[ The second process alters images so reviews are inaccurate.
--[ Some genius figured to write an embedded header altering plugin which locks the files down to only being loadable in a modified instance of Maya.

   The easiest one I know of is to just simply use the generic exr python libraries and write additional header attributes as a post-mel python Frankenscript within the render globals, ugly as heck but works a charm, that way it's technically still an internal Maya "process" completing a rendered image but I'm sure a lot of folks can suggest a more direct and API friendly way of achieving this.

  In the meantime I hope this helps.

Asi Sudai

unread,
Dec 23, 2013, 2:10:43 PM12/23/13
to python_in...@googlegroups.com
I was hoping for a clean way to let MR do the metadata injected and avoid post-mel or a 2nd post-process.
I'm hoping we'll move from MR soon, that's why I don't want to develop more code around it, not cause post-process is a problem.

thanks for replying Evar

Ævar Guðmundsson

unread,
Dec 23, 2013, 3:19:09 PM12/23/13
to python_in...@googlegroups.com
Hehe, cool, have seen so many people get carried away with doing this I'm always retentive on showing people how ( like, seriously, I don't want to get blamed if you then use the method in production and break a delivery frame with it, when you start altering production raw frames as it sounds like is the case here, things tend to get more serious than a forum can accommodate for I'd think )

  If nobody else has pasted by the time I get back in front if Maya I'll send you a plug and play snippet that works in any configuration, super clean for you to evaluate, it's not tricky at all but not a chance I could resite it from memory typing on a phone :)
--
You received this message because you are subscribed to a topic in the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python_inside_maya/Tggunq6Bo0Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/887da10b-69b2-4123-9b18-eac0168fdcc0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Asi Sudai

unread,
Dec 23, 2013, 6:31:35 PM12/23/13
to python_in...@googlegroups.com
After some rethinking I changed my approach a little, reason is,
EXR have a good support for long/custom metadata but others don't have limited support, DPX for example, which we use as well.

So instead of figuring out each format metadata support, I rather dump the metadata in a file or a Database, located where the images/movie are,  I could have infinite amount of metadata without relying on each format ability to store metadata. this is doable in our environment since the rendered files won't get moved from the metadata file. 

I still wish MR would have a built in hook to inject metadata :) but this workaround would help improve on other wider issues as well






To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/AE6A2F8A-0D93-42AC-92F8-9F73ECBBA007%40gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.



--
Asi Soudai | Pipeline Supervisor | Goldtooth Creative Agency, Inc.




aevar.gu...@gmail.com

unread,
Dec 23, 2013, 9:05:14 PM12/23/13
to python_in...@googlegroups.com
  Good answer Asi, got a rant here below, hope it helps ( got too much time on my hands over the holidays apparently)

Asi wrote:
After some rethinking I changed my approach a little, reason is,EXR have a good support for long/custom metadata but others don't have limited support, DPX for example, which we use as well.

  Cool, I’m not sure about how awesome exr is when it comes to long/custom metadata, it’s better than dpx true, don’t get me wrong, the first time .exr was opened to the public I was so excited I learnt to write them out command line pixel by pixel, exr rocks beyond believe but the character limit can get annoying, have you gotten around the 31 character limit yet ( By your description I assume so, but sending the note your way so it doesn’t come as a surprise if it hits )


It gets better though:

Asi wrote:
  So instead of figuring out each format metadata support, I rather dump the metadata in a file or a Database,

  Good call, it’s what most studios do, less risk regarding a frame that's been sitting on the farm for 48 hours and such, however my attitude towards that has always been;
a)
  Why don't those guys hire a render pipeline developer?  a 48 hour render has not been needed since at least 5 years ago, it’s horribly risky and regeneration is near impossible in a tight deadline,  trust me  I’ve seen the biggest frames being shown in cinema these days go through massive farms and there is nothing in the entire world of games/film/commercials which would justify a pipeline where one falls to the grace of the processors and won’t know a thing until days later.
b)
  If done right, this information belongs inside the actual frame as the base location, any database or file system info file should derive it’s information from the image header.  It’s only sensible to include information about things inside things, of course!  It’s only because most people who attempt this completely mess it up that I feel most decision makers rightfully choose the path which has burnt the least and additional databases, info files, excel sheets, and so forth will forever exist.

Asi wrote:
 located where the images/movie are,
Careful;
[pro]
  Easy access to the data for everyone.
[con]
  -You increase the number of files in the project directory, making for more convulated browsing, raising more questions to inherently inquisitive groups of people who really should rather be focusing on their modelling task at hand than be wondering about what the difference between this_shot.mb / this_shot.info / this_shot.dep files are.
  -One can actually foolhardily enough script enough files to get generated into there so it goes beyond 2000 files, loading times from a database environment turns into massive debug sessions to enhance the loading process when all that is happening is unix struggling due to sloppy coding, check out this link, it’s a really good read, suggests only 1020 files per directory which is an absolute most i.m.h.o
  Additional things to back this up below, it’s so crazy I still ran into people in 2013 hitting this limit 😊

  It’s always good to have that stuff accessible but if it’s not inside the image file I’d say keep it away from the general staff ( tell them where it is if they are interested of course but don't complicate their workspace with metadata information they don’t need )

Asi wrote:
  I could have infinite amount of metadata without relying on each format ability to store metadata. this is doable in our environment since the rendered files won't get moved from the metadata file. 
  It’s a good idea, the reason I’m getting into this in further detail is it’s getting to be a bit of an old sport.  Works of course but history has proven requires massive amounts of human resources to support it.
  Alternates do exist out there, for example since you bring up the environment have you considered a context sensitive directory scheme?  Roughly it goes { setenv this_x[1-nearinfinity] for levels of directory depth; that_[x-nearinfinity] for token in naming convention + non identifiable elements}  , It’s a fun one, means you only ever have 1 configuration file that you drop in the root of projects and all metadata is actually generated into the environment based on where you are looking at it from ( hooks into cd on Unix and folder callbacks on Windows, change directory; your environment is different, this can include things all the way down to artist notes even, all still driven from a single algorithm dependant on who is reading it from where, some studios have a rudimentary scheme like this going on but I’ve never seen it fully utilized )

  Then of course, you can always try out the Shotgun Pipeline Toolkit (Formerly Tank) which handles all this at a phenomenal level and since it’s so widespread these days you will be working with a scheme that is consistent across studios, I do think the top of the line with regards to technology we have at hand today is generating metadata into the rendered files ( internally by the renderer as you describe, not a post process, vray and Arnold have the ease of access nailed down ) then push that information towards a hosted Shotgun server as a centralized container.  Must note I have no affiliations with Shotgun development beyond customer level but can verify after having used it since it’s first iteration then develop towards it as Tank became the pipeline toolkit it just makes everything so straight forward.  Keep it in mind, Shotgun rocks for what you are after but mostly so if you manage to embed your metadata.

Asi wrote:
I still wish MR would have a built in hook to inject metadata :) but this workaround would help improve on other wider issues as well

hehe, ok, I fold, just promise to be careful, mental ray picks up custom attributes based on a naming convention, as easy as it gets
  • Experimental: if an attribute with name staring with "meta:" prefix is used (like attribute string "meta:my_key" "my_value"), the value is passed to the image file headers (i.e."my_key" "my_value" pair). This is currently only supported for OpenEXR images. Boolean parameters are converted to integer ones.

  Care to move on to compression?  I can show you how to turn single render into something which halts a 50T server in a few seconds vs. renders without anyone noticing by only toggling a compression setting :)
  exr is awesome to play with, the only format which lets you actually just set that other setting and not lose over 30 hours a day of accumulated time only spent due to number of files in directories and compression schemes, without it the visual effects industry would not have expanded as it has done in the last few years,  it’s a weird thing to claim I know but the efficiency of the format has allowed teams to increase their attentiveness to detail and volume in such a manner that the limitless environment allows for amazing things to be created, the learning curve regarding it though can be a bit tricky as it’s not like other formats.  But hey! it doesn’t have to, it does it’s thing really well :)

  Have a nice holiday ahead
hope your metadata adventures go well
-aeVar

Asi Sudai

unread,
Jan 8, 2014, 1:14:09 PM1/8/14
to python_in...@googlegroups.com
Just landed from the new year holiday to read this detailed replay, very good cover Aevar :)

(in EXR) have you gotten around the 31 character limit
Yes { :( } that's where the external metadata file will help. but 31 might not be enough to store the full path ( as I believe you're pointing out )
1st step - create the metadata file in the same location as the EXR.
2nd Step - adding a pointer ID ( as a metadata ) into the EXR/DPX files. the ID will point into a database entry ( MySQL or Shotgun ) that will point to the full metadata.
This will cover most angles, making sure the EXR's could always be coupled with their metadata.

you can always try out the Shotgun Pipeline Toolkit (Formerly Tank)
As for storing render metadata in Shotgun good or bad,  it's a very interesting/good point:
  • Pro: Metadata entities could be linked to Version entities of the renders.
  • Pro: User can easily find the metadata vie Shotgun web interface.
  • Cons: Shotgun API query are slow ( even more so when it's not a local host )
Currently we do use Shotgun as the main database.
but I'm planning, one day, to migrate data that is less for user-viewing and more for back-end to out local MySQL which is much faster for query.
*migrating could end up in syncing data between the two and not necessarily "hard-migrate"
* a single Shotgun query is fast enough ( for an interactive tool or a process ) and using API3, most times, queries can be bundled into one using "deep-query". But... more than often, there's the need to "attack" shotgun with a multi query which I find are slower than a locally hosted database.

happy new years :) 
Asi



--
You received this message because you are subscribed to a topic in the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python_inside_maya/Tggunq6Bo0Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python_inside_m...@googlegroups.com.

Deke Kincaid

unread,
Jan 8, 2014, 1:42:42 PM1/8/14
to python_in...@googlegroups.com
EXR 1.7 & 2.0 has a 255 characters limit in the metadata.  The issue is if you use more then 31 characters then it makes the files incompatible with applications using the older versions of the library (ie Nuke 7.0 and earlier still used OpenExr 1.6 for flat images).  So most apps have EXR 1.7.x compiled with the LONG_NAMES_FLAG.  As people move to Exr 2.0 that shouldn't be a problem.  We just added an option to truncate the name to 31 characters if you're writing single part EXR 2.0 files if you need the file to be backward compatible.  All multipart files though are full 255 characters in the metadata.

http://lists.nongnu.org/archive/html/openexr-devel/2010-08/msg00017.html

// If an OpenEXR file contains any attribute names, attribute type names
// or channel names longer than 31 characters, then the file cannot be
// read by older versions of the IlmImf library (up to OpenEXR 1.6.1).
// Before writing the file header, we check if the header contains
// any names longer than 31 characters; if it does, then we set the
// LONG_NAMES_FLAG in the file version number.  Older versions of the
// IlmImf library will refuse to read files that have the LONG_NAMES_FLAG
// set.  Without the flag, older versions of the library would mis-
// interpret the file as broken.


--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAP-786epnLhMPQdQWsr8HdD5_hO21xFy0M0otvyU2g1nnpq6Hw%40mail.gmail.com.

Ævar Guðmundsson

unread,
Jan 13, 2014, 11:57:33 AM1/13/14
to python_in...@googlegroups.com

Sounds like you got your hands on something solid there Asi, only have one small point into this one;


On Wednesday, January 8, 2014 6:14:09 PM UTC, Asi Sudai wrote:
  • Cons: Shotgun API query are slow ( even more so when it's not a local host )

  What are you using to benchmark the connection response time?  I ask since I used to claim the same until I measured it and felt quite silly over the results.  I had 3 servers to experiment with, a locally installed staging server, another local server being used for production, and a third hosted server located across the Atlantic.  Turns out the hosted server was giving the best response time!  Shocking and quite weird, I know, so I dug further into it, turned out in the case I refer to the actual response time between a local server and the hosted server on the other side of the planet was marginal and a single query returning a shot entity would actually return in less time than the local one.

  The answer was easy though, The Shotgun server itself was not lagging, it was the difference in code structure in what was being ran against it :)
Reply all
Reply to author
Forward
0 new messages