Fully-funded Post Doctoral Position at InterDigitl, Information Theory for Understanding and Designing Flexible Deep Neural Networks
alexey ozerov
Rennes_France
About InterDigital
InterDigital develops mobile and video technologies that are at the core of devices, networks, and services worldwide. We solve many of the industry's most critical and complex technical challenges, inventing solutions for more efficient broadband networks, better video delivery, and richer multimedia experiences years ahead of market deployment. InterDigital has licenses and strategic relationships with many of the world's leading technology companies. Founded in 1972, InterDigital is listed on NASDAQ and is included in the S&P MidCap 400® index
Job Summary
Running deep neural networks (DNNs) at the Edge and in consumer electronic (CE) devices remains still very challenging due to limited computational and memory resources. Moreover, usually those resources are constantly varying due to other concurrent processes that may start or stop. To optimally use varying resources, one needs so-called flexible DNN models, i.e., models that can optimally exploit all resources available at a given time.
We are targeting the design of such flexible models as follows. A flexible DNN is a network containing several simplified sub-networks of different complexities. This should allow optimally exploiting all available computational resources by executing an appropriate sub-network. However, designing such networks is difficult and time consuming. More precisely, designing an architecture of a usual (non-flexible) DNN is already time consuming, since one needs trying various numbers of layers, various numbers of neurons in each layer, etc. In case of flexible models we are targeting, one needs designing architectures of all sub-networks, which drastically increases the number of tries needed.
In this work we propose to look at the DNNs under the angle of information theory. Indeed, several recent works proposed to study DNNs [1, 2, 3, 4] using the information bottleneck principle [5]. This principle allows obtaining some theoretical bounds of DNN performance by quantifying mutual information between its layers. This will allow finding some near-optimal network simplification rules (going from one sub-network to another) based on theoretical findings, thus avoiding exhaustive cumbersome design of each sub-network. Moreover, we hope that this theoretical work will lead to other new results and findings.
References:
[1] Tishby, Naftali, and Noga Zaslavsky. "Deep learning and the information bottleneck principle." 2015 IEEE Information Theory Workshop (ITW). IEEE, 2015.
[2] Shwartz-Ziv, Ravid, and Naftali Tishby. "Opening the black box of deep neural networks via information." arXiv preprint arXiv:1703.00810 (2017).
[3] Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. "On the information bottleneck theory of deep learning." ICLR, 2018.
[4] Hafez-Kolahi, Hassan, and Shohreh Kasaei. "Information Bottleneck and its Applications in Deep Learning." arXiv preprint arXiv:1904.03743 (2019).
[5] Tishby, Naftali, Fernando C. Pereira, and William Bialek. "The information bottleneck method." arXiv preprint physics/0004057 (2000).
Qualifications
- PhD, in Computer Science, Signal Processing, Machine Learning, Mathematics.
- Strong analytical and problem-solving skills
- Excellent Mathematical / Statistical Skills, Machine Learning and Deep Learning
- Strong knowledge in Information Theory
- Good programming skills and / or simulation tools. SW (C/C++/Python)
- Ability to conduct independent research, propose patents and publications
- Ability to conduct independent research
- Excellent communication skills and ability to work in a team
- ML/AI scientist with a strong expertise and relevant background in distributing model for both inference and training.
- Fluent in English
Location: Rennes, France
InterDigital is committed to a policy of Equal Employment Opportunity and will not engage in or tolerate unlawful discrimination against an applicant or employee on the basis of race, color, religion, creed, national origin, ancestry, citizenship, immigrant status, military status, veteran status, sex, sexual orientation, gender (including gender identity and/or expression), pregnancy, age, physical or mental disability, genetic information, atypical heredity cellular or blood trait, marital status, family status, domestic partner or civil union status or any other legally recognized protected basis under federal, state or local laws, regulations or ordinances. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, compensation, benefits, training, assignments, evaluations, coaching, promotion, discipline, discharge and layoff.
To apply:
Please send your cv and cover letter to Alexey Ozerov Alexey...@InterDigital.com