Scientific Integrity and the History of Deep Learning: The 2021 Turing Lecture, and the 2018 Turing Award
106 views
Skip to first unread message
Schmidhuber Juergen
Jun 29, 2022, 2:45:21 AM6/29/22
to ml-...@googlegroups.com
Hi, fellow machine learning enthusiasts!
Following the great success of massive open online peer review (MOOR) for my 2015 survey of deep learning (now the most cited article ever published in the journal Neural Networks), last year I've decided to put forward another piece for MOOR. I want to thank the many experts (especially those on the connectionists mailing list) who have already provided me with comments on it. Please send additional relevant references and suggestions for improvements for the following report directly to me at jue...@idsia.ch:
https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html
The above is a point-for-point critique of factual errors in ACM's justification of the ACM A. M. Turing Award for deep learning and a critique of the Turing Lecture published by ACM in July 2021. This work can also be seen as a short history of the deep learning revolution, at least as far as ACM's errors and the Turing Lecture are concerned.
I know that some view this as a controversial topic. However, it is the very nature of science to resolve controversies through facts. Credit assignment is as core to scientific history as it is to machine learning. My aim is to ensure that the true history of our field is preserved for posterity.
The lastest version v3 mentions (among many other things):
1. The non-learning recurrent architecture of Lenz and Ising (1920s)—later reused in Amari’s learning recurrent neural network (RNN) of 1972. After 1982, this was sometimes called the "Hopfield network."
2. Rosenblatt’s MLP (around 1960) with non-learning randomized weights in a hidden layer, and an adaptive output layer. This was much later rebranded as “Extreme Learning Machines."
3. Amari’s stochastic gradient descent for deep neural nets (1967). The implementation with his student Saito learned internal representations in MLPs at a time when compute was billions of times more expensive than today.
4. Fukushima’s rectified linear units (ReLUs, 1969) and his CNN architecture (1979).
Thank you all in advance for your help!
Jürgen Schmidhuber
Following the great success of massive open online peer review (MOOR) for my 2015 survey of deep learning (now the most cited article ever published in the journal Neural Networks), last year I've decided to put forward another piece for MOOR. I want to thank the many experts (especially those on the connectionists mailing list) who have already provided me with comments on it. Please send additional relevant references and suggestions for improvements for the following report directly to me at jue...@idsia.ch:
https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html
The above is a point-for-point critique of factual errors in ACM's justification of the ACM A. M. Turing Award for deep learning and a critique of the Turing Lecture published by ACM in July 2021. This work can also be seen as a short history of the deep learning revolution, at least as far as ACM's errors and the Turing Lecture are concerned.
I know that some view this as a controversial topic. However, it is the very nature of science to resolve controversies through facts. Credit assignment is as core to scientific history as it is to machine learning. My aim is to ensure that the true history of our field is preserved for posterity.
The lastest version v3 mentions (among many other things):
1. The non-learning recurrent architecture of Lenz and Ising (1920s)—later reused in Amari’s learning recurrent neural network (RNN) of 1972. After 1982, this was sometimes called the "Hopfield network."
2. Rosenblatt’s MLP (around 1960) with non-learning randomized weights in a hidden layer, and an adaptive output layer. This was much later rebranded as “Extreme Learning Machines."
3. Amari’s stochastic gradient descent for deep neural nets (1967). The implementation with his student Saito learned internal representations in MLPs at a time when compute was billions of times more expensive than today.
4. Fukushima’s rectified linear units (ReLUs, 1969) and his CNN architecture (1979).
Thank you all in advance for your help!
Jürgen Schmidhuber
Reply all
Reply to author
Forward
0 new messages