Familar face recognition like Nest

[CV]

Hi all, I am trying to implement an application similar to Nest camera's familiar face feature. Specifically, it works like this:

- The 1st ever detected face will be a new one and we need to give a name to it. 

- From the next detected face, the app needs to decide whether it is similar to one of the stored face. If not it is a new one and we need to give a new name for this person.

So I think it will involve 2 important tasks:

- Unknown face classification

- Active learning, so that once a new face is added, the application can quickly get a new classifier. 

I've been ignoring the active learning part, i.e., a new classifier is trained at every step of getting a new face, but still the classification is pretty poor. 

Any ideas or suggestions would be greatly appreciated!

Thanks