language dependent lip reading

[yong xu]

Hi,

I am looking into this dataset for lip-reading assisted speech separation.

But i assume that lip reading is dependent on language. What is your comment on this?

Could you help to distinguish the language in the videos? I can see that there are so many languages.

I am only interested in English and Chinese. 

Thanks

yong

[Berkay Inan]

Hello,

I am not affiliated with Google or AVSpeech, but I have some insight about this. I have worked on exact same problem, I used the video only implementation from https://github.com/mpc001/end-to-end-lipreading. You can contact them to obtain weights.

They trained the network using BBC News videos (British English), but in my applications I used the earlier layers to obtain general embeddings which was able to generalize into multiple languages without any modification. I trained and tested it on AVSpeech.

Best,

Berkay Inan

[yong xu]

thanks for your reply. I will try the oxford dataset.

BTW, i notice in the google "Looking to Listen at the Cocktail Party" paper, there is a statistics about the language distribution figure. How do the authors know the language of each video? 

many thanks

best regards,

yong

[Zhang Edison]

Hi Yong,

I think they obtained this information based on YouTube metadata, as you can see in the captions for Fig. 2 -- "with distributions shown in (c) (...language based on YouTube metadata)".

Best,

Yuanhang