Using the tracker

[Sean Kennedy]

Hey guys,

Messing around with the new tracker, works great so far! Definitely seems to be sticking nicely to the features I want to track.

But now, how do we use this info?

Linking a mask point to a tracker center shifts the mask point and makes it unable to be repositioned.

We can add a transform node and link the X and Y values, but how would that work with a roto? Putting the transform after my roto, I'm not seeing the matte moving with the track.

Are we able to only link individual mask points? What about an entire mask as one thing?

Thanks for any pointers you can give!

[omar brown]

Do a lasso select around the mask and then click export in the tracker. The tracker will link pin 1 to the tracker animation and create a corner pin node. In the corner pin node enable pin 1 and connect to roto before the corner pin node and merge you footage.

Long, but logical in procedure.

[MrKepzie]

Hi Sean,

Linking a mask point to a track will indeed make it unable to be repositioned, I'll try to improve this and allow editing, but what should we do, break the link upong edition ? What you can do currently - I guess - is unlinking the mask point, which will still have the animation remaining from the track, but the link will be gone.

I'm afraid you can only link individual points to tracks for now. Once we will implement export to Transform (probably for v1) you will then be able to use this transform for your mask. 

Bear in mind also there's still the export to CornerPin in the meantime

[Omar Brown]

The corner pin works with small rotos that doesn’t leave the frame. When you have a large roto the have to extend beyond the frame for continuous masking, it get cut off and warped at the same time.  See my demo at https://www.youtube.com/watch?v=R3vIQrMGY1s&index=4&list=PLKq_Gnn6tV4pdIrwbDq87VCyqYoQ0Lc1J

--
You received this message because you are subscribed to a topic in the Google Groups "Natron" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/Natron-VFX/V1jTZyBWhmI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to Natron-VFX+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frédéric Devernay]

just modified CornerPin so that only enabled points are drawn.

[Frédéric Devernay]

just fixed the fact that wrong points were enabled.

Le 2 août 2014 à 09:48, Frédéric Devernay <frederic...@inria.fr> a écrit :

> just modified CornerPin so that only enabled points are drawn.
>

> --
> You received this message because you are subscribed to the Google Groups "Natron" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to Natron-VFX+...@googlegroups.com.

[frederic.devernay]

great finding. Filed as issue #142 https://github.com/MrKepzie/Natron/issues/142

[frederic.devernay]

Two additional hints:

- you can average tracks and use the averaged track(s) to feed to a cornerpin

- don't forget that you have the option to invert the cornerpin

[frederic.devernay]

Problem fixed, issue https://github.com/MrKepzie/Natron/issues/142 is closed!

[Sean Kennedy]

Omar I have found an issue with your technique. I will do a quick demo video to show the issue.

Alex, as far as linking mask points to tracks goes, I will put that in the video too. :)

So much easier to show this stuff in action than to type it all out.

[Vegas Editor]

I have notice an issue too.  But would like to see if we both on the same page.

This is your season

Omar Brown

Blessed House Media

--
You received this message because you are subscribed to a topic in the Google Groups "Natron" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/Natron-VFX/V1jTZyBWhmI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to Natron-VFX+...@googlegroups.com.

To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.
To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/5c582347-3a75-473d-b205-05b7cd8756d0%40googlegroups.com.

[Omar Brown]

I retested the process now fully awake. When I linked the RotoNode into the TrackerCornerPin, I notice that by illusion that the corner pin animation is transforming the A channel from the mask input. I think that is whats happen on the youtube demo and my retested one.

[Omar Brown]

The corner pin animation is not transforming the Channel A input from the Mask input. I had overlaid the roto & corner pin in the A input and the Sequence in the B input. I found that the corner pin tracker data offsetting the roto cutting of my footage. The still graphic from my flickr account: https://flic.kr/p/oySGjm.

[Sean Kennedy]

Here's my little video about the tracker. Sorry it's so long, about 20 minutes.

https://www.dropbox.com/s/gjam4pfa9amtbi0/natron_tracking_01.mp4

It's mostly me explaining how I thought it would work and trying out ways it might work, but doesn't.

If I am doing something wrong, and it can already do some of the things I'm trying to do in the video, please let me know how. :)  I want it to work so badly, hahahaha...

[Sean Kennedy]

Just noticed a small mistake in the video. At around 10:30, I put the tracker after the constant and it doesn't move. I should have tried putting it after the roto. I was putting it in the wrong place in that example.

However, I try putting it after a roto in another part of the video, so it still doesn't work as expected.

[Vegas Editor]

Hello Sean,

I saw your video clip and we are on the same page. The release of the tracker was good but need some serious love. You and I are both traditional users of traditional trackers. The tracker in Natron is a little left wing with great potential.

You are far more experience in visual effects than I am, but I have some ideas that might make Natron's 2D tracker 2nd to none. How about we talk shop or brainstorm some ideas for the developer team.

The development team are top R&D Scientist and Computer Vision Programmers.  I don't know if they are actual visual VFX compositing artists. As you know we need certain tools to function and get the jobs done without figuring out a thousand workarounds.

I like were the tracker is going. let's try to get more users and developers involved in Natron.

This is your season

Omar Brown

Blessed House Media

On Aug 2, 2014, at 7:44 PM, Sean Kennedy <mackd...@gmail.com> wrote:

Just noticed a small mistake in the video. At around 10:30, I put the tracker after the constant and it doesn't move. I should have tried putting it after the roto. I was putting it in the wrong place in that example.

However, I try putting it after a roto in another part of the video, so it still doesn't work as expected.

--

You received this message because you are subscribed to a topic in the Google Groups "Natron" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/Natron-VFX/V1jTZyBWhmI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to Natron-VFX+...@googlegroups.com.
To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/7a147598-cda9-40bb-9f5c-cabf609a0782%40googlegroups.com.

[Omar Brown]

An Addendum,

We wouldn't want the Natron tracker to be a clone of other trackers.  Just perform strait forward tracking with some new algorithms, technology and workflow.  I think were the tracker slips up at is in its workflow. Fred did done some coding to fix some of the issues we are having.

This is your season

Omar Brown

Blessed House Media

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/7C94843A-BC9F-4429-9F0B-FB9C2C7FC473%40gmail.com.

[Sean Kennedy]

On Saturday, August 2, 2014 7:33:11 PM UTC-7, Omar Brown wrote:

We wouldn't want the Natron tracker to be a clone of other trackers. 

Well, I think the whole idea behind Natron is to have it look and function as much as possible in the same ways Nuke does. So in fact, the developers may indeed want it to be a clone of Nuke, and thus, Nuke's tracker. Of course, I'm certainly open to hear some ideas about how it should work, but how it works is certainly not my decision to make. I just know if it's not efficient, people will use other software. 

And I'm not too sure tracking needs to be reinvented. Tracking has been perfected over the last 20+ years of visual effects work, I'm not sure how much room there is for new workflows and ideas regarding it. I'd just be happy if I could track a patch in, or parent a roto to a track. :)

 

[Omar Brown]

I am not saying that tracking needs to be reinvented. But Lets compare Blender's Tracker 2D/3D tracker to other trackers for the straight forward workflow. It is not PFTrack or Boujou but I feel its algorithms are equal to a great degree. The developers can do whatever they want with Natron. But lets look at the development of the Natron tracker. It is similar to Nuke 8 without all the options. How many new comers will even think to look at Nuke, let alone its tracker. Natron very much like Nuke, but it still has its own identity and growing culture. I just wanted to take all of our experiences with other compositors and help Natron standout amongst them. This could drive more users, developers and its culture. This is all from a marketing point of view.

Omar

[Frédéric Devernay]

Concerning development of the tracker:
- what we want is to focus on the GUI for now, including the tracking GUI. That's the reason the 2D tracking algorithm is utterly simplistic (it works, it's slow). This tracker actually uses OpenFX, and I think nobody did it this way before us. It's thus very easy to replace the tracking algorithm by another one (for example, we could have a face tracker, people tracker, logo tracker, etc.)
- we've never done compositing, so that what we learn comes mainly from the Nuke tutorials. I guess there are many more use cases
- you, the users, tell us where to go. Getting features from Nuke is usually easier, because we try to follow the same logic. But we don't want to just copy Nuke. We may even diverge from Nuke on some topics in the future (e.g. deep compositing)
- the 2D tracker will be much more flexible when scripting and expressions appear in Natron
- the 3D tracker in Blender seems to be very good, and we don't want to do 3D in Nuke. I think everything 3D should be done in Blender, because that's its job.
- We will probably have a "planar tracker" in the future, but that's where Natron tracking will stop. No Camera tracking probably. We have to interface with Blender instead.

fred

[Sean Kennedy]

Omar, yes, I'm definitely all for Natron (and it's tracker) having it's own identity, but first it needs to work and do the basics. 

Fred, Ok, I see. I agree with you that 3d tracking is best left out of Natron for a while. There are many, many 2d capabilities to tackle first. Same with Planar tracking. No need for that for a while. 

Regarding the current tracker, using it in the same way as Nuke's tracker, it still produces a weird result. In Nuke, you put the tracker node in the tree, then in the transform tab, you choose either matchmove, stabilize, etc... In Natron's tracker transform node, we can only do the cornerpin export data thing. This then creates a corner pin node. So the first tracker node has to be taken out of the tree, and the new corner pin node added in. Then you have to manually enable to1 to activate it. Definitely not the most ideal workflow.

Are there plans to put in the tracker transform options Nuke has? Matchmove, stabilize, etc...?

[Alexandre]

Yes we will add this Transform tab.

We didn't do it by lack of time

In Natron 1.1 we will add expressions and scripting which will allow us also to link the new CornerPin and Transform node exported from the Tracker

directly to the tracks

--
You received this message because you are subscribed to the Google Groups "Natron" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Natron-VFX+...@googlegroups.com.

To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/32f07298-3b04-4b3a-bf0e-231eae3917db%40googlegroups.com.

[Sean Kennedy]

Cool, that's good news.

Now that I understand how it works, still using the corner pin to have a roto follow a track results in a roto that warps it's shape. I guess this is due to the 3 other corners of the corner pin deforming the image. This occurs even though the to2, to3, and to4 "use" boxes are unchecked.

Is there anyway to avoid that waping?

[Omar Brown]

I don't if this will work and I just thought of it. How about copy the Pin 1 animation to the rest of the Pins and don't enable them. I guess they will have the same location as the tracker and see what happens. My graphic markup is at https://flic.kr/p/oiWEPN.

[Alexandre]

Yes we already discussed it with Fred in another thread (can't find it I have like 200+ emails  yet to read…)

I'm currently watching your video and will get back to you afterwards;)

--
You received this message because you are subscribed to the Google Groups "Natron" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Natron-VFX+...@googlegroups.com.
To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/6d608d6e-a08b-4efd-8e09-af4e5fbc099c%40googlegroups.com.

[Sean Kennedy]

In the video, I'm completely forgetting the Transform tab that Nuke uses, so it may not be relevant now. 

[Alexandre]

You seriously made me rofl ! love your comments :P

The tracker works as follow:

For each frame, find in the previous/next frame in the given search window the pattern defined at the current frame.

So if you press "Cancel", it doesn't loose the animation, it just stops. If you restart it, it will pick up the pattern at the current frame as it is.

This is because it is a simple tracker which doesn't maintain an "internal state": Each frame is the same as the others.

Regarding the error threshold to determine if we should stop or not (like you say it is in Blender), we would have to maintain some sort of internal state, or at least compare the pattern again the "first" pattern grab… this is not hard to do though.

You kinda made me laugh when you were struggling with the selection of the roto, do you think selecting something from the panel should change the current tool used in the Viewer GUI to selection mode ?

Regarding the transform applied directly to the tracker's input: we need to implement the Transform tab for that. Currently the tracker node doesn't do any rendering at all, it is just an analyser.

It's interesting that you brought up that the "Link to track…" feature for roto points should work as an offset. It didn't come to my mind and actually it's just doing the opposite: the position of the point is then the position of the track. I will add a checkbox "Link as offset" that will be checked by default.

I guess that for 1.0 we need to implement the Transform tab so that we can properly use the tracker

[Alexandre]

Ho and for the "Link to" option on parameters, this huge list is actually the list of all the parameters in the project that you can link to.

Do you have any clue how could make this look better? Should we instead propose only a subset ? if so which subset ?

[Sean Kennedy]

Ah, it picks up the current frame as it is if you stop and restart it. That is good information to know! I definitely think there should be a STOP button, then.

The error threshold isn't that big of a deal. Blender's tracker just does it so well compared to Nuke's and After Effect's, so I've gotten pretty spoiled by it.

> do you think selecting something from the panel should change the current tool used in the Viewer GUI to selection mode?

No, I was just fumbling around, I think. I'll get used to it.

> Currently the tracker node doesn't do any rendering at all, it is just an analyser.

Ah, this is interesting news! So it's not meant to actually be used in the node tree yet? Should we only be testing the algorithm, seeing what makes it fail, tracking motion blur, testing the UI? That would hange thing a lot, I would stop fighting with it to try and make it work how I want it to, hahahaha...

"Link to Track" with offset, that is definitely needed. How would the checkbox work? I feel like it should just do it by default. In 12 years of professional compositing, I don't think I've ever once needed a roto point to be EXACTLY lined up with a track point. Track points are usually just for motion, not for actual positioning of masks and elements.

I'm wondering, would it help you guys to see see a tracking workflow in other programs? Common uses, the fastest way to work with tracking data, etc?

> I guess that for 1.0 we need to implement the Transform tab so that we can properly use the tracker

Yes please! :)

> for the "Link to" option on parameters, this huge list is actually the list of all the parameters in the project that you can link to.

Do you have any clue how could make this look better? Should we instead propose only a subset ? if so which subset?

Hmmm, I'm not sure. I'd have to be reminded of how Nuke does it. I'm sure there's definitely an better way.

[Alexandre]

The stop button would do the same as cancel. Plus when the tracking dialog is active, you can't click anywhere else than this dialog.

Regarding the "Link to track", I removed the old code and made it an offset instead.

I would like to see any workflow with trackers in other programs if you can;) Just to have clear ideas of what is "must have" and what is optional.

--
You received this message because you are subscribed to the Google Groups "Natron" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Natron-VFX+...@googlegroups.com.
To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/05ba47fd-e521-4f59-87fc-605565501404%40googlegroups.com.

[omar brown]

Alex,

I have been experimenting through the night till morning on how we can use Blender with Natron and possibly avoid some licensing issues. There is a blender developer by the name Michael Krupa who coding the "import/export .Chan" format for blender to/from Nuke. It works quite well between the two apps. Maybe, if possible the transform node can get an import/export .Chan format feature. We can do some extended 2D/3D camera/object tracking in blender and import all of the transformation data in Natron via the transform node, corner pin node and roto node. This way you guys can spend more time on bug hunting/fixing as well as other necessary developments. The development and description of how the blender add-on works is at http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Import-Export/Nuke.

Here are some quick overview of the add-on:

Description:
The *.chan format is used to store camera animation, including location, rotation and optionally field of view.

This addon can import and export chan files using the active object's animation.

Chan file is an ASCII file with parameter values saved in columns, one column per parameter, one line per frame i.e.

1 -0.703427 0.747595 -0.791473 -48.4942 161.026 6.02732 57.840534711
2 -0.681864 0.744656 -0.785405 -48.4888 161.043 6.06889 57.840534711
3 -0.660483 0.74154 -0.779456 -48.4442 161.095 6.06769 57.840534711
4 -0.639921 0.738876 -0.77333 -48.427 161.186 6.04798 57.840534711
5 -0.619503 0.735973 -0.767138 -48.3956 161.284 6.02043 57.840534711
6 -0.599471 0.733561 -0.760774 -48.3782 161.397 5.98574 57.840534711
7 -0.579693 0.731829 -0.754117 -48.3946 161.496 5.96938 57.840534711
8 -0.560836 0.730369 -0.747382 -48.422 161.661 5.90704 57.840534711
9 -0.54107 0.728683 -0.740694 -48.4238 161.742 5.89686 57.840534711

The properties saved and read by this script are: frame location.x location.y location.z rotation_euler.x rotation_euler.y rotation_euler.z angle_y

The file format itself is as simple as can be, but its simplicity is its greatest advantage. It's being used by softwares like Nuke and Houdini, and since it is so simple then writing an import or export tools for it softwares is fast and easy.

"angle_y" stands for vertical field of view. Used for calculation of a camera lens, only applies to the camera animations, while importing/exporting object animation this parameter is skipped.

Instructions:

The script gets the currently active object (works in object mode only) and saves/loads its transformations from or to the simple ASCII file, through the whole animation range (set either on timeline or in the render settings). All you need to do is to select an object and run the script in file->export->Nuke(*.chan) (to export animation) or file->import->Nuke(*.chan) (to import animation).

One thing to be remembered. Chan saves only the raw property values (rotation_x, rotation_y, rotation_z etc.), so you have to mind the rotation order. In other words - the rotation orders during export and import must be the same (both are being set in the file selector while importing/exporting).

Another thing is the camera sensor size and its influence on camera lens. Since 2.61 you can manipulate the sensor size so you can fit a real life cameras (default in blender is 32x18), the best practice in this case is using horizontal fit for camera (Nuke is using it as a default). While importing the camera from *.chan file you have to remember to set the same sensor size as you've had in Nuke (or other software that this camera has been exported).

File names:

It's good to save the chan files with it's rotations order and sensor size stored in a file name (i.e. "camera_for_shot_ZXY_36x24.chan") so you don't have to look for those values in some long forgotten files.

Exporting geometry to other softwares:

If you want to export the objects movement to other softwares via the obj format, you have to save it with the Z-up, Y-forward setting. After loading it to the other software it'll be rotated 90 degrees, but when you apply the chan file it'll jump into its place.

[Francesco Paglia]

Nice video Sean! Just seen it! :)

As I see the tracker already gives a result so it works in terms of data given.

I haven't  properly get the word "rendering" here it just means that actual tracker doesn't process any pixel to make an output isn't it already right? 

> for the "Link to" option on parameters, this huge list is actually the list of all the parameters in the project that you can link to.

Do you have any clue how could make this look better? Should we instead propose only a subset ? if so which subset?

Hmmm, I'm not sure. I'd have to be reminded of how Nuke does it. I'm sure there's definitely an better way.

To better handle the linking GUI we could simply add different levels on the panel like:

|----------------------------------|

|            Node name      |

|----------------------------------|

|         parameter          |

|----------------------------------|

This would be useful in many cases and also will clean a lot the interface when there are many node to search through.

About the linking of roto point.

I think the solution of linking with offset is absolutely the way to go, small fixes can always come.

I was also thinking that a great addition would be to expose the entire roto transformation to link them to a tracker not just the point themselves.    

--
Francesco Paglia

Vfx and Production Supervisor

mobile  +39 347.82.12.473

e-mail   f.pag...@gmail.com

[Sean Kennedy]

That's the perfect solution for the linking parameters! A list of every node you could link it to in the tree, then once you choose one, you get another list of which parameter you want to link directly to. That would clean up the list a lot!! (Unless you have 500 nodes, which I suppose isn't unheard of).

I also agree with the roto linking solution. You should definitely be able to link a mask as a whole, or the individual mask points. Both are very useful.

[Francesco Paglia]

We can even refine the search on the list using the same approach we have with the Tab key.

As soon as we know the name of the node we can just type the first few letter and the filter show the matching node name  

2014-08-06 20:13 GMT+02:00 Sean Kennedy <mackd...@gmail.com>:

That's the perfect solution for the linking parameters! A list of every node you could link it to in the tree, then once you choose one, you get another list of which parameter you want to link directly to. That would clean up the list a lot!! (Unless you have 500 nodes, which I suppose isn't unheard of).

I also agree with the roto linking solution. You should definitely be able to link a mask as a whole, or the individual mask points. Both are very useful.

--

You received this message because you are subscribed to the Google Groups "Natron" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Natron-VFX+...@googlegroups.com.
To post to this group, send email to Natro...@googlegroups.com.
Visit this group at http://groups.google.com/group/Natron-VFX.

To view this discussion on the web visit https://groups.google.com/d/msgid/Natron-VFX/999ea8ee-58a2-4cbd-9bf5-61878eceee66%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Frédéric Devernay]

> Regarding the current tracker, using it in the same way as Nuke's tracker, it still produces a weird result.

...

> Then you have to manually enable to1 to activate it. Definitely not the most ideal workflow.

the wrong points were enabled, and cornerpin was always showing the 4 points whethere or not they were enabled. It was a bug. Fixed a few days ago (after watching Omar's video)

[Frédéric Devernay]

>
> > Currently the tracker node doesn't do any rendering at all, it is just an analyser.
>
> Ah, this is interesting news! So it's not meant to actually be used in the node tree yet? Should we only be testing the algorithm, seeing what makes it fail, tracking motion blur, testing the UI? That would hange thing a lot, I would stop fighting with it to try and make it work how I want it to, hahahaha...

it's meant to be put in your node graph wherever you want. It's just a NoOp/identity node with regards to rendering.

You can then place the CornerPin just below the tracker if you want. I think the GUI issue here is that you don't actually see the link between the cornerpin and the tracker... and the other issue is that there actually is no parameter link, but just a copy.

I filed two new issues concerning this:

Visualize parameter links in the node graph? https://github.com/MrKepzie/Natron/issues/159

Tracker: export to cornerpin should link animations rather than copy them https://github.com/MrKepzie/Natron/issues/160

fred