jitter cause, again
Makc
http://groups.google.com.ua/group/flartoolkit-userz/msg/e3da8c2257daf970
said this:
"...the corners are not determined very accurately. If the threshold
is not VERY close to correct, the
corners can be off by a few (or MORE!) pixels, since the blob will
bloom out or shrink in depending on
how far off the threshold is. If the corners are off by even 1 pixel,
the orientation of the marker
will be wrong..."
I did some tests here to verify this, and I am still *not* convinced
that this is the cause. Basically, I added this code to standard cube
example:
var sq:FLARSquare = detector.getSquare ();
for (var i:int = 0; i < 4; i++) {
var ix:int = sq.imvertex[i].x;
var iy:int = sq.imvertex[i].y;
var c:uint = 0xFFFF0000; bd.setPixel (ix + 0, iy + 0, c);
bd.setPixel (ix - 1, iy + 0, c); bd.setPixel (ix + 1, iy + 0, c);
bd.setPixel (ix + 0, iy - 1, c); bd.setPixel (ix + 0, iy + 1, c);
}
and watched the output for a bit. From just looking at red crosses
jumping around together with my cube, it is easy to think just what
Blair said, but if you try to compare the sequence of images in your
head (or in photoshop :) you will notice a lot of frames with exactly
the same detected square corners but cube still placed differently. I
am attaching two frames for example.
There is no doubt that jitter does come from source image, but it
doesnt seem to be just square corners problem. The question to Blair
is, what *other* things did you modified apart from subpixel corner
detection?
makc
hence the "continue mode" in detector, which is on by default, and
seems to use previous results to compute matrix values in
getTransformMatrix() call. Disabling it seems to have little effect on
jitter, however :(
I also have similar results with sq.sqvertex (which appear to be
substantially different from sq.imvertex but yet just as stable
compared to jitter). So it seems the only suspect is now "lines" array
in FLARSquare... back to reading the code.
makc
On Apr 22, 12:32 pm, makc <makc.the.gr...@gmail.com> wrote:
> It appears original authors were perfectly aware of jitter thing,
> hence the "continue mode" in detector, which is on by default, and
> seems to use previous results to compute matrix values in
> getTransformMatrix() call.
http://www.hitl.washington.edu/artoolkit/Papers/ART02-Tutorial.pdf
(page 15)
It's kinda funny how this doesnt solve jitter in flash version at all,
just makes it different.
> So it seems the only suspect is now "lines" array
> in FLARSquare
detection. You can see that in getSquareLine () method of
FLARSquareDetector - sqvertex array is calculated as intersection of
lines. Therefore, the key to removing jitter should be in better line
fitting method, or if you go with refined corners as Blair suggests,
you still have to update lines to match new corner coords.... back to
thinking.
makc
this paper,
http://www.sonycsl.co.jp/person/rekimoto/papers/apchi98.pdf It is
called "Matrix: A Realtime Object Identification and Registration
Method for Augmented Reality", and markers are referred to as "2D
matrix markers". Just a year before 1st matrix movie, seems like a
strange correlation :)
Blair MacIntyre
>> It appears original authors were perfectly aware of jitter thing,
>> hence the "continue mode" in detector, which is on by default, and
>> seems to use previous results to compute matrix values in
>> getTransformMatrix() call.
>
> I actually found a proof of that here
> http://www.hitl.washington.edu/artoolkit/Papers/ART02-Tutorial.pdf
> (page 15)
>
> It's kinda funny how this doesnt solve jitter in flash version at all,
> just makes it different.
It's not surprising. The original one didn't work very well either.
It's all a form of smoothing, similar in effect to all the stuff
people here have tried. In the end, you can't recreate lost data.
>> So it seems the only suspect is now "lines" array
>> in FLARSquare
>
> In fact, page 16 says ARToolKit does 4 lines fitting, not 4 corners
> detection.
That's only referring to how they determine if it's a square (they
walk the edge, looking for segments that form lines, and throw out any
blobs that have more or less then 4).
> You can see that in getSquareLine () method of
> FLARSquareDetector - sqvertex array is calculated as intersection of
> lines. Therefore, the key to removing jitter should be in better line
> fitting method, or if you go with refined corners as Blair suggests,
> you still have to update lines to match new corner coords.... back to
> thinking.
I'll have to look at the code (again) to see if I believe this. We
took the output corners from the ARToolkit and refined them with the
original image. Because the ARToolkit pose-estimator sucks (it's
slow, and is unstable, which is probably what you are seeing), we
actually use (now) a different one (also from OpenCV). The
combination gave VERY stable tracking.
HOWEVER, using a better pose estimator REQUIRES having a calibrated
camera. If you try using better, more accurate/stable pose estimators
with bad data (e.g., images taken from cameras where the calibration
data is not actually for the current camera), they perform very badly
(sometimes failing, or giving horrible results).
For those who don't know, the calibration files tell the system what
the internal properties of the camera are. These include the field of
view, where the optical center of projection of the image is (for
typical cameras, the center of projection of the lens will not fall
exactly in the center of the image sensor), and an estimate (typically
expressed as the coefficients of a polynomial) describing the barrel
distortion of the lens (these are the distortions that cause lines to
curve as they approach the edge of the image). These are the typical
parameters for a normal camera.
The values change if you change the zoom OR the focus of the camera!
Even for focus, they change slightly, and for cheaper cameras, if you
return to a previous setting, the values might be different (because
the lens has physically shifted in it's seating).
The one "good" thing about the ARToolkit optimization-based pose
estimator is that it will at least work (for some definition of
"work") when the camera you are using does not really correspond to
the calibration (.dat) file you are using (which, in the case of FLAR,
is almost always the case!).
Line fitting only works if the camera is calibrated, since that
calibration is used to remove barrel distortion and other other
distortion from the image (making it rectilinear, and making the edges
straight). If you are not able to do this (again, since we don't have
calibrated cameras, we can't properly undistort the image), you can't
do anything useful with the edges.
BTW, the original ARToolkit camera calibration approach was also
pretty poor.
So, what some of the ongoing discussion about jitter is probably
seeing relates not just to the quality of the corners, but to the
LOCATION in the image of the marker. If the camera is not calibrated
and there are ANY distortions that are different than the calibration
file. If you move the marker in image to the left/right (by even a
pixel) you may get a slightly different result.
I also suspect that, since the camera calibration is not correct, that
the optimization is hitting different local minima based on it's
starting conditions; it's getting "plausible" results that are within
the error tolerance of algorithm, but are slightly different each time.
Blair MacIntyre
invented this approach; perhaps he did, though, it's been so long
since I actually looked this up. I thought the ARToolkit predated,
or at least was parallel, to Rekimoto's work. However, I could be
wrong.
BTW, I believe Rekimoto's group created the tracking technology for
the PS3 Eye of Judgement game, for those who care about trivia.
People in the computer vision community have done vision based
tracking of squares for a while, though. The big deal with Rekimoto's
work and the ARToolkit was in using it for AR.
eric socolofsky
one somewhat OT thing i just thought about -- it's truly awesome that
saqoosha spent all this time porting NYARToolkit into AS3, but...the
real refinement of the engine is (i imagine) happening in the original
C++ codebase, in the original ARToolkit. FLARToolkit isn't likely to
see those upgrades without someone to keep an eye on the ARToolkit
codebase, and to upgrade continuously.
a better solution for FLARToolkit is to loop it into the relevant
segments of the ARToolkit codebase via Alchemy. this has the double
benefit of performance increase and closer tie-in with the developers
who *really* know what's going on, and are actively developing. i'm
definitely not qualified to do this...but hopefully someone will step
up to the plate sometime in the near future. i know saqoosha is
investigating possibilities with Alchemy and PixelBender, but not
really sure what route he's taking.
back to on-topic, in the meantime, makc, seems like the most rewarding
route to go would be to control jitter with easing, on the client side
(rather than within the engine). while that kind of correction is
perhaps a bit more computationally-expensive than fixes within the
engine, and also likely results in less responsiveness (as it smoothes
all motion, including jitter), it probably won't make too much of a
difference in the application speed, and will have a substantial
positive impact on the appearance of the model positioning/
transformation.
yet another thing to add to the list for FLARManager.
-e
makc
On Apr 22, 9:50 pm, eric socolofsky <e...@transmote.com> wrote:
> one somewhat OT thing i just thought about -- it's truly awesome that
> saqoosha spent all this time porting NYARToolkit into AS3, but...the
> real refinement of the engine is (i imagine) happening in the original
> C++ codebase, in the original ARToolkit.
are at revision 2 from 2004, with actual content dated 2001, so I
doubt there are any substantial changes.
> back to on-topic, in the meantime, makc, seems like the most rewarding
> route to go would be to control jitter with easing, on the client side
> (rather than within the engine).
I learned reading sources today, it is nowhere near easy to modify
this "pose estimator" algorithm, but I still have an idea or two to
try. I will report back positive results (or none).
makc
> ...I still have an idea or two to
this problem caused primrily by fluctuating size of white square after
application of fixed threshold to grayscale image, then maybe adapting
threshold to average local brightness in grayscale image would result
in more stable blob and reduce jitter. Well, it did not really helped
to reduce jitter, but it did improved detection under changing
lighting, check it out:
http://www.youtube.com/watch?v=_JBdLq0QT6g
the downside of adaptive filter is, however, that it notices much more
details, so I had to shrink search area towards last detected marker,
and only search in whole image when confidence goes down or nothing is
detected... (this in itself appears to be nice offspring of my little
jitter research)
eric socolofsky
about this the other day, and it's on my list to add to FLARManager
(adaptive thresholding). good to hear you had some success! would
you mind sharing some source?
thanks,
-eric
eric socolofsky
On Apr 23, 2009, at 10:31 AM, makc wrote:
makc
On Apr 23, 9:04 pm, eric socolofsky <e...@transmote.com> wrote:
> i had a conversation with someone with previous AR / CV experience
> about this the other day, and it's on my list to add to FLARManager
> (adaptive thresholding). good to hear you had some success! would
> you mind sharing some source?
>
> thanks,
> -eric
>
(you have commit rights, right?), basically, non-breaking changes to
detectors. at the moment, filter is declared there as
private var _tobin_filter:FLARRasterFilter_BitmapDataThreshold;
if you could change it to
private var _tobin_filter:IFLARRasterFilter_RgbToBin;
and add public getter/setter for it in detectors, I could just wrap my
stuff into class implementing IFLARRasterFilter_RgbToBin.
This change would also require you to add function setThreshold
(i_threshold:int):void to IFLARRasterFilter_RgbToBin interface, which
imho is small price for being able to set all kinds of custom filters
on the fly.
Blair MacIntyre
on a Graz web page) and pull some of these features from it. It's
GPL'd, too; so, none of these changes could be folded into the
commercial version, but it would help the community. They implemented
adaptive thresholding, vignette correction, and other things.
eric socolofsky
you can apply for rights on your own, they're pretty open about it.
from this page:
"The person who wants to be a commiter, please send your username and
password or the SHA1 hash of your password to beintera...@gmail.com
. "
http://www.libspark.org/wiki/WikiStart/en
that's what i did, after prompting from saqoosha, and shindo
yoshihiro, the libspark organizer, was very open to contribution.
blair -- thx for the ARToolkitPlus suggestion. will look into it.
makc, if you do the same, let's keep communicating here so we don't
duplicate work.
-e
makc
On Apr 23, 10:37 pm, Blair MacIntyre <blair.macint...@gmail.com>
wrote:
> on a Graz web page) and pull some of these features from it. It's
> GPL'd, too; so, none of these changes could be folded into the
> commercial version, but it would help the community. They implemented
> adaptive thresholding, vignette correction, and other things.
>
studierstube tracker page,
http://studierstube.icg.tu-graz.ac.at/handheld_ar/images/stbtracker_adaptivethresholding.jpg
lo and behold:
http://xs538.xs.to/xs538/09175/stbtracker_adaptivethresholding_result760.jpg
back to ARToolkitPlus, according to description there, "ARToolKitPlus
can do dynamic thresholding by looking at the marker content (pattern)
and taking the average between the darkest and brightest pixels. If no
marker is found the threshold value is randomized." - this looks like
something that can be dan with FLARToolkit without no modification at
all - "on the client side", as Eric calls it.
eric socolofsky
apps for too long ;)
interesting they randomize threshold; i was thinking more of
incrementing it and finding a good range, or possibly taking a tree
approach, undershoot then overshoot then undershoot a little less then
overshoot a little less, zeroing in on a good value / range.
Thomas Brault
Thanks you makc !
---------------------
Thomas Brault
thomas...@gmail.com
http://www.tomysnockers.net
+33.6 43 33 86 64
Makc
On Tue, Apr 28, 2009 at 12:46 PM, Thomas Brault <thomas...@gmail.com> wrote:
> Very impressive result, but i don't realy understand your modifications,
> what filter did you apply exactly ? Can you zip your source and share it ?
>
sorry I was far far away from the internet for this whole week, I will
do this shortly.
@eric, I've got commit rights to spark, but I could not get in touch
with saqoosha about this. I assume I could commit any way, because
changes I need (a setter for to-bin filter) are not breaking, but it
certainly would be nice to have his agreement beforehand.
Saqoosha
sorry for long time no reply.
similar modifications are already applied my local version.
so, your changes are welcome. thank you!
but if you wanna commit big changes,
it's better to create anothre branch.
I didn't experience any team development.
any suggestion are welcome.
--
Saqoosha
s...@saqoosha.net
http://saqoosha.net/
Makc
http://makc3d.wordpress.com/2009/05/05/flartoolkit-adaptive-filter-experiment-sources/