pointcloud v3 OpenEXR feature request

[tom.c...@gmail.com]

Would it be possible in a future version for the EXR export to include the RGB data as well? Right now having to export the EXR files for depth and then a separate texture pass to get the colour and then recombining them in the compositing software.

OpenEXR supports an insane number of data channels per frame, and a preferred configuration would be to have the R G B channels assigned to the color data, Z for the depth data and the black areas without any data into an alpha channel.*

*That last one is especially helpful, as right now in the EXR nearest is black, but also completely missing data is 100% black, so without an extra step to luma matte out the 0.0 regions they are places right at the near plane when converted back into z space.

[tom.c...@gmail.com]

Oh and one final request that might be far fetched, but a position pass would be incredible too. Using the world cords of the multiple aligned cameras but packed into the EXR, it would allow the RGB color points of each sensor to be positioned based on their world position in Nuke/Fusion/blender etc

[Brekel]

Interesting idea, having used Nuke myself in the past I understand where you're coming from.

Currently I'm using an image file wrapper library that I don't think supports additional channels for OpenEXR (or other) files, I'll have to look into that.

Op woensdag 23 juni 2021 om 20:36:22 UTC+2 schreef tom.c...@gmail.com:

[Brekel]

Well guess what.... :)

Turned out it wasn't too hard to do an implementation with the OpenEXR SDK directly.

Latest version now supports writing RGB, Alpha, Raw Depth and XYZ channels with various compressions algorithms.

Sensor data like type, transform and lens calibration data is saved as custom header attributes.

From what I can see using the Natron app all things seem to look properly saved, although I couldn't test if the sensor transform is correct (matrix could be incorrectly transposed in theory).

Greets,

Op woensdag 23 juni 2021 om 20:48:07 UTC+2 schreef Brekel:

[tom.c...@gmail.com]

Great! Though it seems the depth channel is flipped horizontally from all the other channels

[Brekel]

What brand/type of sensor was that with?

Op donderdag 24 juni 2021 om 20:11:17 UTC+2 schreef tom.c...@gmail.com:

[tom.c...@gmail.com]

Azure Kinects

[Brekel]

Thanks, will check.

Azure Kinect's are one of the few sensors that mirror their color channel and I didn't check if I should correct for that in the exporter (tested with a Kinect v2 during dev)

Op donderdag 24 juni 2021 om 20:21:38 UTC+2 schreef tom.c...@gmail.com:

[tom.c...@gmail.com]

Yeah trying out the EXRs in nuke now it seems the layers are a bit mislabeled, where black magic fusion was able to find the RGB channels it needed me to manually point to the Depth and world pass channels, nuke doesn't seem to be able to parse them correctly. I think it's reading the world position layer as the default and getting confused

[Brekel]

These are the channel definitions from my code:

header.channels().insert("R", IMF::Channel(IMF::HALF));

header.channels().insert("G", IMF::Channel(IMF::HALF));

header.channels().insert("B", IMF::Channel(IMF::HALF));

header.channels().insert("A", IMF::Channel(IMF::HALF));

header.channels().insert("Depth", IMF::Channel(IMF::HALF));

header.channels().insert("X", IMF::Channel(IMF::FLOAT));

header.channels().insert("Y", IMF::Channel(IMF::FLOAT));

header.channels().insert("Z", IMF::Channel(IMF::FLOAT));

Seemed to work in Natron (which is all I have to test), if the names should be defined differently just let me know.

Op vrijdag 25 juni 2021 om 00:51:43 UTC+2 schreef tom.c...@gmail.com:

[tom.c...@gmail.com]

yeah so from what I understand that is making a unique channel for each instead of a layer containing a set of channels. So this is what the channels of something I rendered out in fusion and brought over to nuke to read back looks like.

So the rgba items are all rgba.red rgba.green rgba.blue rgba.alpha which together makes a layer called RGBA

Depth is actually the "Z" channel, but it's in the layer depth. 

Then in the case of fusion, it seems to write it's worldpass into other.

Natron I guess is just globbing everything together anyway and sorting it out, but that layer grouping is how different passes are isolated.

[Brekel]

Would if be possible for you to share an EXR file (not originating from my app) with those layers/channels so I can try to analyze it and reproduce it's structure?

Op vrijdag 25 juni 2021 om 01:20:43 UTC+2 schreef tom.c...@gmail.com:

[tom.c...@gmail.com]

a bit further it seems nuke doesn't have a standardized layer name for world pass, and different rendering packages name it differently, but usually the axis are posX, posY, posZ so not to be confused with other channels that might use x/y/z like motion vectors or smart vecs etc

[tom.c...@gmail.com]

lol google won't let me upload an exr here for security purposes

[tom.c...@gmail.com]

Can I post a drivelink?

https://drive.google.com/drive/folders/1UvnzS6mWy6p8xKnV_L2onWm7Evjg53aN?usp=sharing

[Brekel]

Thanks, will have a look at that tomorrow.

(you can also email me directly of course at any time)

Op vrijdag 25 juni 2021 om 01:36:48 UTC+2 schreef tom.c...@gmail.com:

[tom.c...@gmail.com]

added "NUKEsampleEXR.00001.exr" which is one I generated by using a scanline renderer to make a bunch of channels, and then creating a custom world pass layer and then rendering an EXR with all the layers included.

"NUKEsampleEXR_Multiview.00001.exr" is a really crazy one that has a second perspective rendered into cam1 multiview 

[tom.c...@gmail.com]

also djv is a good piece of software for viewing openexr sequences without having to load them up into a program or drop the crazy money for shotgun rv 
https://darbyjohnston.github.io/DJV/

[Brekel]

Thanks!

Op vrijdag 25 juni 2021 om 01:59:04 UTC+2 schreef tom.c...@gmail.com:

[Brekel]

So here's the contents of the files, Nuke packages Z & positions in two layers, BlackMagic does not.

Nuke:

A

B

G

R

depth.Z

pos.posX

pos.posY

pos.posZ

BlackMagic:

A

B

G

R

Z

posX

posY

posZ

I checked and I can replicate either one in my app.

Would one be easier to work with than the other in production use?

Op vrijdag 25 juni 2021 om 10:36:52 UTC+2 schreef Brekel:

[tom.c...@gmail.com]

my guess is everyone follows the foundry's lead on layer configurations so it would be the most compatible. 

[Brekel]

That was my gut feel and apparently also the consensus I got from asking the question on social media, so I'll match Nuke's.

Op vrijdag 25 juni 2021 om 16:36:28 UTC+2 schreef tom.c...@gmail.com: