Some basic understandings about immersive and kartaverse workflows

[Gary Jaeger]

Equipment:
Camera body: Canon R5

Lens: RF5.2mm F2.8 L Dual Fisheye

Atmos Ninja V+

I’ve gotten myself up to speed on the basics of Immersive video generation and delivery for Apple Vision Pro using the kartaverse workflow in Resolve. I’m sure I have a lot to learn on the subtleties including adjusting parallax etc, but for the time being I just wanted to check some of my findings and questions here.

The _injected files play on vision pro native either in their little square window or filling the field of view with a soft vignette when the “Immersive” button is pressed. They also play in Moon Player, though differently than the native playback.

a) the apple native playback seems to offer the most parallax feel, though it’s always as a fisheye (distorted) and never goes to 180° field of view even when the Immersive button is pressed, in that case it's always has a soft matte.

b) when using Moon Player and set to 180° and Mono, I do get a 180° field of view without the fisheye, but the parallax feels lessened. 

c) when setting the SuperST Mapper from Fish_Eye to Flat, the fisheye effect is removed in the native AVP player, but now the effective field of view is diminished. This would otherwise be the best option, but where did the rest of the image go? I understand that spatial quality will be very low as it approaches the edges, but is there any way to map it w/o the fisheye but keep the full 180?

So I guess I’m curious how to get the full 180, non-fisheye experience. Is that just a limitation of resolution or aspect ratio? At the end of the day do you need a 2:1 image per eye to get that effect? Here I’m thinking of the sample immersive videos on the AVP. Obviously higher end capture, but beyond that I think there is something else fundamentally different about that workflow.

Any info appreciated. 

Cheers and thanks.

-gary

[Nick Brazzi]

Here's what I do:

-Use the Kartaverse SuperSTMapper to convert the Fisheye view from the camera to a normal equirectangular Side-by-Side 3D 180 Frame.

-Use other Kartaverse tools in the editing process
-After editing, export that to Prores HQ with full HDR PQ settings

-Use Andrew Hazelden's "Spatial Metadata GUI" tool to convert that to MV-HEVC.

-ZIP that MV-HEVC, then airdrop it to the Apple Vision. 

-On the apple Vision, use the files app to unzip. 

Why do the zip and unzip? Because if you airdrop the video file directly, it adds it to your Apple Photo Library and that doesn't help me. I need it in the normal file storage.

-On the Apple Vision,  use the OpenImmersive app to play the video. Chose "Open from Files" and navigate to the MV-HEVC file

https://apps.apple.com/us/app/openimmersive/id6737087083

OpenImmersive is made by Acute. They also license their player to many of the streaming apps on the Apple Vision (Immersive Bliss, etc.). So, it gives you a similar experience to those streaming apps.

[Gary Jaeger]

This is great info, thank you

[Gary Jaeger]

So interesting how all the players play back the content slightly differently. Openimmersive seems great though it does seem to squish the vertical FOV. I'll dig into it more, probably user error on my side

[Nick Brazzi]

I haven't noticed any squishing, but I don't have the most trained eye. I'm curious - what is your timeline resolution?

[Gary Jaeger]

Well that was just a quick test I ran just now .CRM > VR Utility > Spatial Metadata GUI. But I did look at some of my other renders from Resolve and it seemed like it was doing the same. More testing needed. 

[Nick Brazzi]

I was curious about your resolution.

I don't know if this is related to your issue, but I recommend you check out this video, starting at 7:05 for some interesting notes about resolution. 

https://youtu.be/GKul6PTLwlU?feature=shared&t=425

Based on this, when I edit, I'm using 8192x4320. And I actually skip the steps at 8:14 about changing the resolution and scaling on export. Since I'm not delivering to Youtube, I think the native 8192x4320 will be fine.

[Gary Jaeger]

Yeah, same 8192x4320 timeline resolution. I've been doing the scale on export but maybe I'll try not doing that. 

[Gary Jaeger]

Nick, are you using the same lens? RF5.2mm F2.8 L Dual Fisheye? If so can I ask what settings you're using in Spatial Metadata GUI

[Gary Jaeger]

fwiw, here's where I am currently 

Image Projection: Fisheye

Stereo 3D Format: Side by Side

Primary Eye: Right

Horizontal FOV: 190

Interaxial: 60

Horizontal Disparity: 0 (not sure about this one)

Manual Bitrate: 360

Thanks again. Your input has been most helpful. Actually OpenImmersive app was a big key. 

[Nick Brazzi]

Yes. I'm using the same lens.

Aside from the bitrate, I'm pretty sure I never touched any of the settings. I believe these are all the default values and they work just fine:

Image Projection: 180VR

Stereo 3D Format: Side By Side

Primary Eye: Left

Horizontal Field of View: 180

Interaxial Lens Separation: 64mm

Horizontal Disparity Adjustment: 0

Now that you have me thinking about it, I guess the IPD for the lens is 60. Maybe I should try that.

Also, that lens does capture 190-degree FOV. BUT - after the Kartaverse STMap processing, is it still 190? Or is it conformed to a standard 180? I'll admit, you have me curious to try those settings when I have time tomorrow.

But, with those default settings, things have been looking pretty good for me.

[Gary Jaeger]

Yeah, that's why I was using 60 and 190. The other thing I've been trying is setting the timeline to 59.94 and the motion to Optical Flow. Actually a really big improvement. I'm also going to try a Topaz Video up to 60 to compare. Cranking up the bit also really cleaned up the final final. Obviously a lot larger but worth it. It got rid of a lot of buzzing around the edges.