Megaface Challenge

[Scott Winterringer]

http://megaface.cs.washington.edu/

I'm curious to see how well this holds up to the challenge.

This challenge is based around a very large database

[Brandon Amos]

Hi Aaron,

I remember you sent in a few OpenFace PRs a while ago. Did you ever try OpenFace on the MegaFace benchmark? If so, I don't expect it to (currently) be very competitive with the current industry techniques, but I'd be interested in your implementation so we can track our progress on it as we improve.

-Brandon.

[Brandon Amos]

Hi Aaron,

Thanks, great info! We're slowly (but actively) looking into
improving the performance when trained with VGG.

Peng Liu found an issue with my inception tower matrix sizes
in late May at https://github.com/cmusatyalab/openface/issues/142
that I corrected on June 3 in this commit:
https://github.com/cmusatyalab/openface/commit/49fffb36710086e0a0540d194fd53e7954fd563c
Re-training with the fixed version didn't improve our accuracy.

I'll ping you if we find anything else that helps us improve
our accuracy when training with the VGG dataset.

-Brandon.

* Aaron Nech :: 2016-07-07 13:01 Thu:
> Hi Brandon,
>
> We did an initial testing (with your NN4 model) and OpenFace performed very
> poorly (higher than hand-engineered features, but lower than all other conv
> net approaches). The model dived very quickly in performance as the number
> of distractions increased.
>
> We also tried training OpenFace on a data set much larger than the
> pre-trained networks were trained on (we're releasing this data set towards
> the end of summer) and it also performed poorly (although better than the
> previous results). It was also below other algorithms trained on smaller
> amounts of data. You also obtained no major improvement in accuracy when
> trained on VGG ( https://github.com/cmusatyalab/openface/issues/103 ).
>
> Therefore we think this indicates an issue inside OpenFace, but we're not
> sure what. I suspect there is a small bug in the somewhere which is
> limiting the performance of the model. Do you have any intuition what this
> could be?
>
> You're totally welcome to try the Megaface challenge as OpenFace changes.
> If you don't have a copy of the distraction dataset, fill out a form on our
> website and you'll get credentials for it soon after.
>
>
> Aaron Nech
> Computer Engineering

[Fede Ruiz]

Hey Brandon, it appears that MegaFace has made their dataset available for download here: http://megaface.cs.washington.edu/dataset/download.html

Its 1 million faces (65 gbs). That's gigantic. I don't know what you're looking for in terms of its license, but regardless it'd be super interesting to see how Openface perfoms with the neural net trained on it. 

[Brandon Amos]

Hi Fede,

 

it'd be super interesting to see how Openface perfoms with the neural net trained on it.

I agree, it would be great to incorporate megaface's huge dataset into OpenFace's network training. However megaface doesn't contain identity information and can't be used with our training that requires identity information. It would be interesting to explore ways to train with labeled and unlabeled data, but it's currently an open research problem.

-Brandon.

[Fede Ruiz]

Oh, I see.

So where's the difficulty right now in training the network with a larger database? I'm assuming making some sort of script that scrubs social networks for pictures with identity information wouldn't too hard to make, so I'm guessing the difficulty arises because of licensing issues?