What is the status of keypoint detection and keypoint matching: machpoint (gsoc2007) and feature_matching (gsoc2008)
Harry van der Wolf
I've been searching my archives but I'm a bit lost (or actually: completely lost).
During gsoc 2007 Zoran Mesec created/worked on matchpoint: the patent free keypoint detector. As far as I'm aware matchpoint works fine, but still has problems dealing with transparency masks.
True or wrong? Anyone who does know, please explain. Any relevant info that I didn't ask here is welcome too.
During gsoc 2008 Onur Küçüktunç created/worked on feature_matching to be used in combination with matchpoint. Is this now (relatively) mature or not? Can we use it? As far as I know this still hasn't been integrated into the main Hugin trunk or has it? (and am I a complete moron for missing that?)
Finally:
Combination of matchpoint and match-n-shift: Is this now a working keypoint detection/matching combination as such or does match-n-shift "just" use autopano for the keypoint matching? (I'm almost sure about the latter, but maybe I also miss some details here). I know match-n-shift does much more, but I'm now only referring to keypoint detection/matching.
hoi,
Harry
Tom Sharpless
I too would like to know the status of the various control point
finders associated with Hugin. And I don't think Harry and I are
alone in this. Is it possible that someone who knows (maybe that
means a committee??) could publish an authoritative and intelligible
summary of this topic? I could contribute a bit on Autopano-sift-c
but am wholly ignorant of the others.
-- Tom
Bruno Postle
>
>During gsoc 2007 Zoran Mesec created/worked on matchpoint: the patent free
>keypoint detector. As far as I'm aware matchpoint works fine, but still has
>problems dealing with transparency masks.
>True or wrong? Anyone who does know, please explain. Any relevant info that
>I didn't ask here is welcome too.
If I understand Zoran correctly, matchpoint is very good at
detecting scale invariant features, but the feature descriptors
generated are not as good as those from SIFT or SURF.
I'm not sure that this is very important - Current feature matching
tools such as those in Onur's project, panomatic or autopano-sift
work entirely by comparing these descriptors, but for panoramas we
can compare the geometrical relationships between points too.
>During gsoc 2008 Onur Küçüktunç created/worked on feature_matching to be
>used in combination with matchpoint. Is this now (relatively) mature or not?
It worked for me, but I have no idea why it wasn't integrated. Even
if the results are not so good, any patent-free control point finder
is better than none.
It is implemented as part of the hugin GUI which makes it impossible
to script around. I would very much like a command-line version
(equivalent to the 'autopano' tool from autopano-sift-C).
>Combination of matchpoint and match-n-shift: Is this now a working keypoint
>detection/matching combination as such or does match-n-shift "just" use
>autopano for the keypoint matching?
match-n-shift uses a combination of autopano-sift-C and
align_image_stack. There is a --matchpoint option which substitutes
the 'generatekeys' part of autopano-sift-C, but it doesn't work
because of the alpha channel bug. Even so match-n-shift still uses
the 'autopano' part of autopano-sift-C for feature matching.
[later...] I just added an ImageMagick workaround to match-n-shift
--matchpoint and now it seems to work ok. Also match-n-shift runs
ptoclean as a final step, so it kind-of does the 'geometrical' check
needed.
I hope Pablo will add to this thread, he mentioned that between
the various projects (matchpoint, panomatic, autopano-sift-C,
feature_matching) there is all the code necessary to paste together
a complete patent-free solution.
Otherwise, a command-line version of Onur's feature matching would
be enough to allow me to make match-n-shift 'patent-free'.
This whole subject just needs somebody to pick up the pieces and put
them together.
--
Bruno
Zoran Mesec (zoran.mesec@gmail.com)
On Feb 26, 9:51 pm, Bruno Postle <br...@postle.net> wrote:
> On Wed 25-Feb-2009 at 14:19 +0100, Harry van der Wolf wrote:
>
> >During gsoc 2007 Zoran Mesec created/worked on matchpoint: the patent free
> >keypoint detector. As far as I'm aware matchpoint works fine, but still has
> >problems dealing with transparency masks.
> >True or wrong? Anyone who does know, please explain. Any relevant info that
> >I didn't ask here is welcome too.
>
Wrong. Matchpoint has a great potential for the future but it needs
futher improvements on the descriptor part for public/general use.
Feature detection(my GSOC 08 project) consists of steps:
a.)detecting features - this is done OK in Matchpoint
b.)creating descriptors for detected points in step a). Matchpoint:
this is not accurate enough at the moment. We improvised with the
desciptor, because we had to pick one that is patent free. But I did
not work out as we expected - accuracy is poor.
Accuracy was measured using a test suite and benchmarked to other
descriptors&detectors with the same suite.
Onur's part(Feature Matching) comes in after step b.) Feature matching
takes the descriptors and matches them. It is a separate process
(descriptors are read from an xml file).
> If I understand Zoran correctly, matchpoint is very good at
> detecting scale invariant features, but the feature descriptors
> generated are not as good as those from SIFT or SURF.
>
>
> It worked for me, but I have no idea why it wasn't integrated. Even
> if the results are not so good, any patent-free control point finder
> is better than none.
(called matchpoint at the end).
In order to do that the matchpoint descriptor needs to be improved. If
that is done and the suite is created then we will have what we want.
However, I do not have resources(knowledge&time) now to deal with the
algorithms and I was hoping someone else has. I can assist him, but
can't do all myself, sory. Work that needs to be done is not too
difficult but it requires some effort.
Perhaps we could make this task a mandatory conditon for wannabe GSOC
09 participants so that they can show programming skills(but the
theory and algorithm needs to be determined in advance by us)
best regards,
Zoran Mesec
Bruno Postle
>
>Because we would like to see both projects integrated into one suite
>(called matchpoint at the end).
>In order to do that the matchpoint descriptor needs to be improved. If
>that is done and the suite is created then we will have what we want.
..but fedora and debian ship hugin with a warning message instead of
a control point finder because of the patent situation.
Actually, from my small amount of testing yesterday, matchpoint
combined with ptoclean is 'good enough' already. If we can't have
'perfect' then we should release what we have and let others get
annoyed enough with it to fix it.
i.e. can somebody please hack feature_matching into a command-line
tool and we can merge it into the trunk with the GUI parts disabled.
This won't break anything but will enable further work.
--
Bruno
Harry van der Wolf
I downloaded Onur's gsoc2008_feature_matching branch again and it builds fine on OSX with cmake (that's a plus but Onur worked on Mac too so that might be a good reason).
In the "images" tab when clicking the "Create Controlpoints" button I have three extra options:
- Feature Matching with Linear Search
- Feature Matching with Single KDTree
- Feature Matching with Multiple KDTree
The first option finds nothing. The second and third option make Hugin crash.
Anyone who tested this had same or other results on Linux or windows?
Second to that: I checked the code and it seems quite isolated from the rest. Onur's code hasn't changed since he stopped working on it and the "branch" in src/hugin_base/algorithms" and some connecting header files with minor changes can be "taken out" and combined with matchpoint (as Zoran already mentioned) or as a separate feature_matching tool. The only point where it matches/interferes with current Hugin stuff is the Assistantpanel.cpp, AutoControlPointCreator.cpp and ImagesPanel.cpp and these were only for integration and can be abandoned when choosing for a stand-alone tool.
Unfortunately, I don't have the knowledge: I don't even know how to make a main() {blahblah} around Onur's stuff and to compile it. The reference to main() was before '92 when I was young (and even more handsome) and could build&compile curses based stuff in good old plain C.
Harry
Michael Galloway
> Well,
>
> I downloaded Onur's gsoc2008_feature_matching branch again and it builds
> fine on OSX with cmake (that's a plus but Onur worked on Mac too so that
> might be a good reason).
> In the "images" tab when clicking the "Create Controlpoints" button I have
> three extra options:
> - Feature Matching with Linear Search
> - Feature Matching with Single KDTree
> - Feature Matching with Multiple KDTree
>
> The first option finds nothing. The second and third option make Hugin
> crash.
> Anyone who tested this had same or other results on Linux or windows?
>
i can have a look at it on linux this weekend.
-- michael
Harry van der Wolf
I did a quick try to build it on Ubuntu but failed as it seems I have 3 versions (1_33, 1_34 and 1_35) of boost on my server. The package manager did a bad job I suppose. I need to go back to 1 version and try again (but wasn't very enthousiastic actually)
Harry
Bruno Postle
>
>I downloaded Onur's gsoc2008_feature_matching branch again and it builds
>- Feature Matching with Linear Search
>- Feature Matching with Single KDTree
>- Feature Matching with Multiple KDTree
>
>The first option finds nothing. The second and third option make Hugin
>crash.
'Linear and Single' both worked from me on fedora at the end of the
SoC, 'Multiple' crashed but Onur said that we only needed 'Single'.
This was with Terry Duell's 'two photo' tutorial images:
http://hugin.sourceforge.net/tutorials/two-photos/
The code still builds, but I haven't done any recent testing.
--
Bruno
Pablo d'Angelo
Bruno Postle schrieb:
> On Wed 25-Feb-2009 at 14:19 +0100, Harry van der Wolf wrote:
>> During gsoc 2007 Zoran Mesec created/worked on matchpoint: the patent free
>> keypoint detector. As far as I'm aware matchpoint works fine, but still has
>> problems dealing with transparency masks.
>> True or wrong? Anyone who does know, please explain. Any relevant info that
>> I didn't ask here is welcome too.
>
> If I understand Zoran correctly, matchpoint is very good at
> detecting scale invariant features, but the feature descriptors
> generated are not as good as those from SIFT or SURF.
Yes, the interest point detector in matchpoint works ok, but the
descriptor is currently not good.
> I'm not sure that this is very important - Current feature matching
> tools such as those in Onur's project, panomatic or autopano-sift
> work entirely by comparing these descriptors, but for panoramas we
> can compare the geometrical relationships between points too.
>
>> During gsoc 2008 Onur Küçüktunç created/worked on feature_matching to be
>> used in combination with matchpoint. Is this now (relatively) mature or not?
>
> It worked for me, but I have no idea why it wasn't integrated. Even
> if the results are not so good, any patent-free control point finder
> is better than none.
I remember that the feature detection code just contains duplicate
copies of the matchpoint code, so we would have the same code in two
locations: src/hugin_base/algorithms/control_points and src/matchpoint
The code in matchpoint should be made a library and linked from within
hugin.
I will have some time for hugin development this month (wife and baby
are traveling, I'm stuck at home...), and I will look into these issues
and make sure that we will have patent-free solution that is well
integrated into hugin.
> It is implemented as part of the hugin GUI which makes it impossible
> to script around. I would very much like a command-line version
> (equivalent to the 'autopano' tool from autopano-sift-C).
>
>> Combination of matchpoint and match-n-shift: Is this now a working keypoint
>> detection/matching combination as such or does match-n-shift "just" use
>> autopano for the keypoint matching?
>
> match-n-shift uses a combination of autopano-sift-C and
> align_image_stack. There is a --matchpoint option which substitutes
> the 'generatekeys' part of autopano-sift-C, but it doesn't work
> because of the alpha channel bug. Even so match-n-shift still uses
> the 'autopano' part of autopano-sift-C for feature matching.
>
> [later...] I just added an ImageMagick workaround to match-n-shift
> --matchpoint and now it seems to work ok. Also match-n-shift runs
> ptoclean as a final step, so it kind-of does the 'geometrical' check
> needed.
>
> I hope Pablo will add to this thread, he mentioned that between
> the various projects (matchpoint, panomatic, autopano-sift-C,
> feature_matching) there is all the code necessary to paste together
> a complete patent-free solution.
So we have:
matchpoint (see above)
autopano-SIFT (patented algorithm (although the SIFT patent seems rather
weak and might not apply if the SIFT descriptor is used together with
another interest point detector).
panomatic (SURF, also patent. However, the interest point detector alone
is not really patented, so we just need another feature descriptor).
The panomatic code is very nice and clean, and contains all the required
matching stages, including a ransac step for throwing out outliers
(which probably currently doesn't work with fisheye images and very wide
rectilinear ones). I have already created a separate executable of the
keypoint generator, it should be quite easy to do so for the matching
stage, too.
> Otherwise, a command-line version of Onur's feature matching would
> be enough to allow me to make match-n-shift 'patent-free'.
Actually, something like ptoclean should really be part of the core
functionality in hugin.
Hmm, so should we make your perl modules a requirement for hugin?
Actually, I have also started a python wrapper around the core hugin
libs (using boost-python). I also hope that this will be finished the
end of this month, but I'm not sure if I can finish both the feature
matching and the python interface. (as I mostly write code during the
day, so its sometimes not that attractive to do the same in the evening,
too).
ciao
Pablo
Bruno Postle
>
>> Otherwise, a command-line version of Onur's feature matching would
>> be enough to allow me to make match-n-shift 'patent-free'.
>
>Actually, something like ptoclean should really be part of the core
>functionality in hugin.
>Hmm, so should we make your perl modules a requirement for hugin?
It wouldn't be a problem if it was just Linux. Also I'm sure
somebody could write something better than ptoclean, there is also
ptscluster which takes a different approach that should also be
adopted.
The two other features from match-n-shift that need to be in any
'best of' control-point finder are switching automatically to
align_image_stack for bracketed series and using conformal
projections for the feature descriptors.
--
Bruno
Onur Kucuktunc
If you need anything about feature matching part, I'll be happy to help.
All my reports, and the detailed information about the code changes in
this branch (gsoc2008_feature_matching) can be found at
(http://code.google.com/p/google-summer-of-code-2008-pano/downloads/list).
> I remember that the feature detection code just contains duplicate
> copies of the matchpoint code, so we would have the same code in two
> locations: src/hugin_base/algorithms/control_points and src/matchpoint
Matchpoint code in
(hugin_base/algorithms/control_points/FeatureExtractorZoran) wasn't
actually just a duplicate of (matchpoint). It was meaningless to read
the xml output of Zoran's matchpoint in the project, so I decided to
change the code in a way that it returns the pano object with the
keypoints -without writing into any xml file. However, it'll be better
to make matchpoint a library, as Pablo says.
> 'Linear and Single' both worked from me on fedora at the end of the
> SoC, 'Multiple' crashed but Onur said that we only needed 'Single'.
'Linear' was just for test purposes. 'Feature Matching with a Single
kd-tree' method was the one in my proposal. And 'Multiple' was an
experimental method that improves the accuracy with less computation
in theory.
Best,
Onur