Understanding HoG output

[bricklemacho]

Hi, new to group, new to image processing starting to explore HoG.
Looking for a python implementation and discovered
skimage.feature.hog().

My plan to run skimage.features.hog over some positive/negative images
and use this to train a svm classifier from scikits-learn. I am
trying to understand the output hog before I proceed further. When
I run skimag.feature.hog() it over a region of interest it appears to
returns an array. How do I interpret this array? Is there a way to
reshape the array to see what it was like before it was flattened or
that doesn't make any sense? Can I plot the descriptor returned in
any meaningful way?

Also when I choose to visualise the HoG often where I expected to see
vertical line dominate, say on the edge of builds, the line drawn
often appears to be more dominant at the 45 deg. Is this expected as
the line drawn is really just the sum of all surrounding orientations
for the "cell"?

Thanks in advance,

Michael.
--

[Brian Holt]

Hi Michael, 

Hi, new to group, new to image processing starting to explore HoG.

Looking for a python implementation and discovered

skimage.feature.hog().

Glad to hear someone's using HoGs!

My plan to run skimage.features.hog over some positive/negative images

and use this to train a svm classifier from scikits-learn.   I am

trying to understand the output hog before I proceed further.    When

I run skimag.feature.hog() it over a region of interest  it appears to

returns an array.  How do I interpret this array? 

My best suggestion would be that you take a look at the comments written in `hog.py`.  Each part quotes the relevant section from the Dalal Triggs paper and then under that implements the code required to do the job.  Others on this list may have different views, but I'd be reluctant to try to 'interpret' the flattened descriptor.  

 Is there a way to reshape the array to see what it was like before it was flattened or that doesn't make any sense?  Can I plot the descriptor returned in

any meaningful way?

Yes, you can reshape the array (but that can be a bit tricky), or you could modify the hog function to return the unraveled descriptor and ravel it yourself later if you need it.

Also when I choose to visualise the HoG often where I expected to see

vertical line dominate, say on the edge of builds, the line drawn

often appears to be more dominant at the 45 deg.  Is this expected as

the line drawn is really just the sum of all surrounding orientations

for the "cell"?

A line is drawn for each gradient bin with an intensity proportional to the magnitude of that gradient.  So, the 'star' shape you see for each cell is just the superimposition of all of these lines. You should expect to see dominant lines perpendicular to lines in the image (parallel to the gradient). Also remember that the default is to use 9 bins, so it may be that the 45degree dominant line you see is the closest approximation to horizontal.  You can test this out by trying 8 bins instead of 9. 

I hope it helps, feel free to ask any more questions.

Regards
Brian

On 9 February 2012 06:37, bricklemacho <brickl...@gmail.com> wrote:

Thanks in advance,

Michael.
--

--
He is no fool who gives what he cannot keep to gain what he cannot lose. 
 - Jim Elliot.

[Emmanuelle Gouillart]

Hi Brian,

thanks a lot for the explanations! I must say that when I had a look at
the HoG function following Michael's post, I had quite a hard time
understanding how this function was working. Your explanations do help;
do you think it would be possible to write a meaningful example using
HoGs for the example gallery? That might give a good starting point in
order to use this function. If you don't have the time to write the
example yourself, any hints on the possible contents would also help.

Cheers,
Emmanuelle

[Stéfan van der Walt]

On Fri, Feb 10, 2012 at 9:17 AM, Emmanuelle Gouillart
<emmanuelle...@nsup.org> wrote:
> understanding how this function was working. Your explanations do help;
> do you think it would be possible to write a meaningful example using

And Brian obliged...

http://scikits-image.org/docs/dev/auto_examples/plot_hog.html

Stéfan

[Emmanuelle Gouillart]

Wow, that was quick ;-) ! Thanks for the pull request, will have a look
at it right now.

Emmanuelle

[bricklemacho]

> My best suggestion would be that you take a look at the comments written in
> `hog.py`.  Each part quotes the relevant section from the Dalal Triggs
> paper and then under that implements the code required to do the job.
>  Others on this list may have different views, but I'd be reluctant to try
> to 'interpret' the flattened descriptor.

Doh! didn't think to look at the code. Based on this suggestion I
have had a look at the code, good level of commenting in the code.

> Yes, you can reshape the array (but that can be a bit tricky), or you could
> modify the hog function to return the unraveled descriptor and ravel it
> yourself later if you need it.

Thanks for the idea.

> A line is drawn for each gradient bin with an intensity proportional to the
> magnitude of that gradient.  So, the 'star' shape you see for each cell is
> just the superimposition of all of these lines. You should expect to see
> dominant lines perpendicular to lines in the image (parallel to the
> gradient). Also remember that the default is to use 9 bins, so it may be
> that the 45degree dominant line you see is the closest approximation to
> horizontal.  You can test this out by trying 8 bins instead of 9.

Ahh.. I guess I was expecting some sort of "edge" like in the edge
detector. Makes more sense now.


Also, thanks for the simple example.

[bricklemacho]

On Feb 10, 9:45 pm, Brian Holt <bdho...@gmail.com> wrote:
<snip>

> A line is drawn for each gradient bin with an intensity proportional to the
> magnitude of that gradient.  So, the 'star' shape you see for each cell is
> just the superimposition of all of these lines. You should expect to see
> dominant lines perpendicular to lines in the image (parallel to the
> gradient). Also remember that the default is to use 9 bins, so it may be
> that the 45degree dominant line you see is the closest approximation to
> horizontal.  You can test this out by trying 8 bins instead of 9.

Okay, I am still having problems understanding visualisation of the
HoG. When I plot the visualisation it seem to make sense. It is
still not clear why I don't see vertical/perpendicular lines. If you
look at http://tinypic.com/r/dy0fud/5 I would expect to see the pole
of the street light to show dominant perpendicular lines, but they
appear to be more horizontal than vertical. Similar stuff happens if
I use buildings, the "edge" of the building, clearly visable in the
gradient images, ends up as either tending towards the 45deg (or
135deg). The parameters used for the picture were: Orientations 9,
Pixesl Per Cell16x16, Cells per Block 3x3.

I will trace the code to try to get a better understanding of the
visualisation output, but would appreciate if anyone has any advice
or explanation of my misunderstanding of the visualisation.

Also what is the best why to link to images in a post?

Thanks,

Michael.
--

[Tony Yu]

On Thu, Feb 16, 2012 at 6:32 AM, bricklemacho <brickl...@gmail.com> wrote:

On Feb 10, 9:45 pm, Brian Holt <bdho...@gmail.com> wrote:
<snip>
> A line is drawn for each gradient bin with an intensity proportional to the
> magnitude of that gradient.  So, the 'star' shape you see for each cell is
> just the superimposition of all of these lines. You should expect to see
> dominant lines perpendicular to lines in the image (parallel to the
> gradient). Also remember that the default is to use 9 bins, so it may be
> that the 45degree dominant line you see is the closest approximation to
> horizontal.  You can test this out by trying 8 bins instead of 9.

Okay, I am still having problems understanding visualisation of the
HoG.  When I plot the visualisation it seem to make sense.  It is
still not clear why I don't see vertical/perpendicular lines.   If you
look at http://tinypic.com/r/dy0fud/5  I would expect to see the pole
of the street light to show dominant perpendicular lines, but they
appear to be more horizontal than vertical.  Similar stuff happens if
I use buildings, the "edge" of the building, clearly visable in the
gradient images, ends up as either tending towards the  45deg  (or
135deg).   The parameters used for the picture were: Orientations 9,
Pixesl Per Cell16x16, Cells per Block 3x3.

I don't actually understand HoG that well, but this result, I believe, is expect. Assuming HoG does what the name suggests, you should expect the lines to align with the direction with the largest gradient. So, if the pole of the street light is a dark gray, and the background is white, then the lines should be perpendicular to the pole since this is the direction of the highest gradient. 

[Brian Holt]

Hi Michael,

>
> I don't actually understand HoG that well, but this result, I believe, is expect. Assuming HoG does what the name suggests, you should expect the lines to align with the direction with the largest gradient. So, if the pole of the street light is a dark gray, and the background is white, then the lines should be perpendicular to the pole since this is the direction of the highest gradient.
>

Tony's answer is spot on here.  Perhaps you expect that the HoG image
should look like the gradient image? Instead, what the descriptor is
really aiming to capture is the direction (that's the 'oriented'
part), that the gradients go. So assuming an image where the left half
is black and the right half white, there would be a vertical of
horizontal (or as close as possible, depending on the number of bins)
lines in the HoG image.

> I would expect to see the pole of the street light to show dominant perpendicular lines, but they appear to be more horizontal than vertical.

This is what I was referring to when I talked about the number of
bins. If you look carefully at the HoG image, you'll notice that
there are vertical lines in some places (like at the black billboard),
but there are no perfectly horizontal lines. The closest
approximation to horizontal is maybe a 20deg line. That's because
there are 9 bins. If you tried this with 8 bins, you should see some
horizontal lines.

Hope it helps
Brian

[bricklemacho]

Hi Brian, and Tony.

Thanks both for your response. I hope my newbie terminology is not
making this more confusing.

> Tony's answer is spot on here.  Perhaps you expect that the HoG image
> should look like the gradient image? Instead, what the descriptor is
> really aiming to capture is the direction (that's the 'oriented'
> part), that the gradients go. So assuming an image where the left half
> is black and the right half white, there would be a vertical of
> horizontal (or as close as possible, depending on the number of bins)
> lines in the HoG image.

Okay, makes sense, the direction of the change would be horizontal
direction if the line in the orignal image was vertical. This is how
I originally interpreted the visualisation provide by the skimage-hog
algorithm.

The problem was when I comparing this to the visualisation from the
Dalal & Triggs paper. In the paper the R-HOG descriptor seems to
show a different relationship, when visualised the dominant
orientations appear to be parallel to lines in the original image. If
you have access to the Dalal & Triggs paper, I am basing my
expectations on Figure 6(e), top of page 8. I don't have access to
the original Dalal & Triggs visualisation code so I will reread the
paper to make sure I am comparing like with like.

> This is what I was referring to when I talked about the number of
> bins.  If you look carefully at the HoG image, you'll notice that
> there are vertical lines in some places (like at the black billboard),
> but there are no perfectly horizontal lines.  The closest
> approximation to horizontal is maybe a 20deg line.  That's because
> there are 9 bins.  If you tried this with 8 bins, you should see some
> horizontal lines.

If I change play with the bins I do end up with horizontal lines. I
suspect what is being visualised is different, or being interpreted
differently, to what the Dala & Triggs paper is visualisation. I
just have to understand the different visualisations.

Thanks again for your help.

Michael.
--

[bricklemacho]

Thought I would follow up. I have sorted out the visualisation
expectations. I have been able to modify the visualisation output/
plot of the D&T paper. SOme simple maths can make the orientations
"parallel" if I so desire. It also appears that the D&T paper plot
the orientations after block normalisation which also accounts for why
I was seeing different in gradients. Anyway, I have a better
understanding now, so thanks everyone for your help.

Now to work out how to calculate HoG descriptor for a region rather
than a whole image.

Michael.
--

[Brian Holt]

Hi Michael,

I'm glad you've got it all sorted out.

An issue report has been raised regarding the HoG around a point (actually a list of points). It seems to me that there are 2 options.

1 we could compute the HoG over the whole image and then copy the descriptor values around a given point and return. This requires very little change to the existing code, but will be inefficient for those only requiring sparse keypoints.

2 move the heavy lifting into another function and call it on small image patches. Good for sparse keypoints, bad for dense.

So it depends on what you want really. Quick win is number 1.

Cheers
Brian