Tracking 2D points for mmSolver, for when the Maya plate is over scanned

[Michael Karp]

Because most studios over scan their unwrap plate renders (with a larger Maya film back and larger pixel count), the 2D tracks from Maya, to mmSolver may not match with the e.g. 110% over scan

Let's discuss which one of these solutions makes sense to the forum. If you have a better method, please share.

• Method A: 2d track the unwrap plate in 3de. This method is very simple. 3DE Film back does not need to be correct and 3DE distortion should be at the default zero value.
• Method B: Export a 2.5D mel script from 3DE to Maya (the regular way) and then Reparent the 2.5D locators into the Maya root, then use the mmSolver Convert to Marker tools.
• Method C: Create a complicated tool in mmSolver to offset the 3DE 2.5D track by the e.g. 1.1 or 1.2 times over scan. This method can get complicated to write, lots of over scan and LCO values to think about for the pipeline developer.

[David Cattermole]

Hello Michael,

My preferred method is an alternate of "Method C". Although "Method A" will work if you're asked to track a Comp-manipulated plate, and I consider this an easy fall-back technique.

Let me know if you can see any problems.

Method C Alternate:

1) Track points with the original raw plate (and lens distortion)

2) Select track points, run 3DE script "Copy Tracks (mmSolver)" - this will create a temp .uv file, with undistorted tracking point data.

3) Open "Load Marker" UI (temp file path will be filled out automatically), set correct camera and press "Load" button.

4) Select the "markerGroup" node (the transform node icon with a Marker cross icon in it), then set the "overscan" value (in the channel box) to 1.1. ("overscan" atttribute was added in release v0.1.1) **

5) Done! The markers will align to an overscanned plate with 110% overscan.

Notes:

- This is different from "Method C", because we don't need to convert 2.5D points.

- The camera in Maya should have an overscanned film back (and an overscanned image plane)

- If some points use "Method A" or "Method C", just create a new "markerGroup" node, with a different overscan value. All markers under the "markerGroup" will use the overscan value of it's parent. ***

- This technique does not need to worry about Lens Center Offset (LCO), as we're dealing in normalised image coordinates (.uv file).

------------------------------------------

My brief summary of the other methods are below.

Method A

Pro:

- Simple / fool-proof - it should always work.

Con: 

- Assumes lens distortion won't change.

- It makes me feel wrong tracking an undistorted plate.

Method B

Pro:

- ???

Con:

- Requires users to manually save a file path (with Export 2.5D Points)

- Removes "enable" status from marker (marker is assumed to be enabled always when converting from a transform).

- Any meta-data possible from a .uv file is not possible - this removes the possibility of updating your points automatically if the lens distortion changes.

Method C

Pro:

- Tracking markers can be made on original plate, with original lens distortion, no special plate generation is required.

- If lens distortion changes, very little work has been lost (re-copy points to Maya).

Con:

- Users may break their lens distortion in 3DE and wonder why it doesn't line up in Maya anymore.

- User needs to set a single overscan attribute to the specific value used on the print / shot.

- If using "2.5D points", same "Con" points as Method B (see Method C Alternate above for a technique to use .uv files)

** Without writing studio-specific code (to look up a database value for example), I'm not sure the best approach to allow users to preconfigure this value to eliminate users needing to set the value. Any ideas are welcome! The current best idea is to use an environment variable specified in the ".mod" file for a site-specific global override. 

*** Currently there is no tool to "Add Marker Group". Until such a tool is made, you can "trick" the Load Marker tool into creating a new Marker Group node by un-parenting the the current marker group from the camera, using the Load Markers tool, then re-parenting the original Marker Group node. You may also re-parent individual markers to different Marker Groups by simply middle-mouse dragging in the Outliner (as long as the Translate X/Y is locked).

David

--
You received this message because you are subscribed to the Google Groups "Maya MatchMove Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maya-matchmove-s...@googlegroups.com.
To post to this group, send email to maya-match...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/maya-matchmove-solver/CABzSVOzqP-b0Z%3D_2SLmTCVLXDq7FufGgkWkVK%3DrNQV5JuZh5Kg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Karp]

Thanks David!

Here are a few more real life scenarios that might cause trouble while importing 2D tracks from 3DE, to mmSolver.

1. Does the 3DE 2D exporter take rolling shutter into consideration? Dneg and Rodeo are now rendering about 5% to 10% of their unwrap plates with Rolling Shutter GridWarp correction  I'll test this week, but if 2D points don't line up, we might need to talk to Rolf or use the crude (but effective) method A.
2. Prime Focus always uses 10% over scan for spherical and 20% for anamorphic (easy with mmSolver). But Rodeo has a different percentage over scan for every solve. So one solve might have e.g. 7% over scan in x, 8% in y and then we would need to divide the unwarp pixel count by the original scan pixel count, to determine specific over scan percentage for mmSolver 2D import. And will it matter if the over scan percentage is asymmetrical x/y? Which do we use as a percentage, x or y? We'll have to test. (The reason that Rodeo does this is so that no extra, unnecessary pixels are rendered in the over scan. For example 20% over scan suffers 44% more pixels to render).
3. Double Negative has a proprietary system where the plate is only unwarped in OpenGL hardware, in the Maya viewport. No software rendering and the over scan is already figured out by the proprietary system. But there is another issue there, which is that their clients will commonly supply plates with ten different resolutions (Red, Alexa, anamorphic/spherical, LCO, extraction pivoted from the bottom of frame Common Bottom Line, 35 mm film, etc., for the same show) and all 3DE solves get converted to a common delivery extraction in Maya. This means that the original scan plate in 3DE is almost always a very different format than the extracted delivery plates used in Maya. Keeping track of the various conversions and applying them to imported 2D points got complicated, so in Maya we would Parent Constrain the original unextracted plate camera to the Extraction Delivery Maya camera, so that the 2D points would line up. This is can be done with a variation of Method B, which is where a second camera is used as a "projector" of 2D points for the main camera.

To view this discussion on the web, visit https://groups.google.com/d/msgid/maya-matchmove-solver/CABCsN26UUkKCMfHEYgE0CxR3nBXekmf85ud0VTEXakipGVAOUw%40mail.gmail.com.

[David Cattermole]

Hello Michael,

I've responded inline.

David

On Sat., 9 Feb. 2019, 8:53 pm Michael Karp <michael...@gmail.com wrote:

Thanks David!

Here are a few more real life scenarios that might cause trouble while importing 2D tracks from 3DE, to mmSolver.

1. Does the 3DE 2D exporter take rolling shutter into consideration? Dneg and Rodeo are now rendering about 5% to 10% of their unwrap plates with Rolling Shutter GridWarp correction  I'll test this week, but if 2D points don't line up, we might need to talk to Rolf or use the crude (but effective) method A.

No, good point. That makes things more difficult in 3DE because the rolling shutter must be applied at the same depth as the rolling shutter was exported at. I'll think about the best approach to this. I don't think Rolf needs to get involved, I'm confident we have the tools required for this already.

1. Prime Focus always uses 10% over scan for spherical and 20% for anamorphic (easy with mmSolver). But Rodeo has a different percentage over scan for every solve. So one solve might have e.g. 7% over scan in x, 8% in y and then we would need to divide the unwarp pixel count by the original scan pixel count, to determine specific over scan percentage for mmSolver 2D import. And will it matter if the over scan percentage is asymmetrical x/y? Which do we use as a percentage, x or y? We'll have to test. (The reason that Rodeo does this is so that no extra, unnecessary pixels are rendered in the over scan. For example 20% over scan suffers 44% more pixels to render).

Asymmetrical and symmetrical overscan is not yet tested, only uniform overscan has been tested. However I would guess that symmetrical (x and y) overscan should work as long as the horizontal overscan value is used.

Asymmetrical overscan (bottom left, bottom right and top left and top right coordinates) is another problem to think about.

1. Double Negative has a proprietary system where the plate is only unwarped in OpenGL hardware, in the Maya viewport. No software rendering and the over scan is already figured out by the proprietary system. But there is another issue there, which is that their clients will commonly supply plates with ten different resolutions (Red, Alexa, anamorphic/spherical, LCO, extraction pivoted from the bottom of frame Common Bottom Line, 35 mm film, etc., for the same show) and all 3DE solves get converted to a common delivery extraction in Maya. This means that the original scan plate in 3DE is almost always a very different format than the extracted delivery plates used in Maya. Keeping track of the various conversions and applying them to imported 2D points got complicated, so in Maya we would Parent Constrain the original unextracted plate camera to the Extraction Delivery Maya camera, so that the 2D points would line up. This is can be done with a variation of Method B, which is where a second camera is used as a "projector" of 2D points for the main camera.

The method you explain of handling multiple plate resolutions sounds unfortunate for the MatchMove department. In such a situation "Method A" should work. However if the cropping, LCO and re-position were all done with (projector) cameras, and therefore a film back / FOV / Projection Matrix could be extracted somehow (by selecting the two cameras?), then this difference could be computed and applied to the Marker translate attributes, while retaining metadata and "enable" status of the Markers. 

I'd call this tool "Transfer Markers", as it would move Markers between cameras retaining the screen-space position. I have no plans to write such a tool in the immediate future. Such a tool would require a bit of thinking of use cases and testing. Actually such a tool may already exist... It's called "re-parent", I'd expect the concept is the same, just the GUI and underlying mechanics would differ.

[David Cattermole]

Hello Michael and group,

Thinking about this more, I believe a workflow for the "Transfer Markers" tool I describe above is now possible - albeit without a single fancy UI.

Now that the Save Markers (and Copy Markers) tool exists, this transfering 2D Markers between cameras should be possible by saving markers from one camera, and loading the markers on the new camera (or the same camera).

The Load Markers "Use Embedded Overscan" check-box can also be used to maintain the relative field of view between the cameras, or not.

I've described the workflow on the issue:

https://github.com/david-cattermole/mayaMatchMoveSolver/issues/13

The "real" Transfer Markers tool is still fairly low-priority unless anyone thinks it is important(?).

I assumed the original problems you described have been solved once the "Use Embedded Overscan" check-box was added to the Load Markers UI.

David

--

David Cattermole