The Deep Learning textbook is a resource intended to help studentsand practitioners enter the field of machine learning in generaland deep learning in particular.The online version of the book is now complete and will remainavailable online for free.
If you notice any typos (besides the known issues listed below) or have suggestions for exercises to add to thewebsite, do not hesitate to contact the authors directly by e-mailat: feed...@deeplearningbook.org
Founded in 1917, Deep Springs College is a unique institution of higher learning. The educational program is built upon three pillars: academics, student self-government, and manual labor. The school is located forty miles from Bishop, California on an isolated cattle ranch in Deep Springs Valley.
Between twelve and fifteen students are admitted each year. Each receives a full scholarship; the college covers the costs of tuition, room, and board for every student offered admission. In exchange, Deep Springs students are expected to dedicate themselves to lives of service to humanity. Alumni have gone on to exemplify this ideal in a variety of fields, including politics, science, journalism, academics, agriculture, medicine, law, business, and design.
Together with the Telluride Association, founded in 1911, these compose The League of Nunnian Schools. The League aims to support existing and developing Nunnian projects in America and around the world. If you have questions, requests, or would like to otherwise contact the League, please write to com...@deepsprings.edu.
Students at Deep Springs participate in running the college itself in ways that are not possible at larger institutions. The Student Body exists as a discrete, self-governing entity within the College, tasked with governing itself and working with the staff and faculty in the day-to-day operation of the college.
The Student Body convenes at least once a week to deliberate and take action on issues concerning community life, labor, academic planning, disciplinary matters, election of student office holders, and more. Students are elected to positions of leadership within the Student Body and the larger community. Elected positions include Student Body President; Labor Commissioner, who assigns students labor positions and is in charge of making sure the labor program runs smoothly; and Student Trustee, one of two students who sits on the Board of Trustees of the College as full voting members.
Since Deep Springs is only a two-year program, students do not often see the full outcomes of their decisions. But just as they are the recipients of decisions made by students before them, so they, in turn, are obligated to try to make the best choices possible for students who follow. This chain of responsibility, from one generation of students to the next, continually reinforces and reinvigorates the gravity students at Deep Springs take in governing their college.
Deep learning is a subset of machine learning that uses multilayered neural networks, called deep neural networks, to simulate the complex decision-making power of the human brain. Some form of deep learning powers most of the artificial intelligence (AI) applications in our lives today.
While supervised learning models require structured, labeled input data to make accurate outputs, deep learning models can use unsupervised learning. With unsupervised learning, deep learning models can extract the characteristics, features and relationships they need to make accurate outputs from raw, unstructured data. Additionally, these models can even evaluate and refine their outputs for increased precision.
Deep neural networks consist of multiple layers of interconnected nodes, each building on the previous layer to refine and optimize the prediction or categorization. This progression of computations through the network is called forward propagation. The input and output layers of a deep neural network are called visible layers. The input layer is where the deep learning model ingests the data for processing, and the output layer is where the final prediction or classification is made.
Another process called backpropagation uses algorithms, such as gradient descent, to calculate errors in predictions, and then adjusts the weights and biases of the function by moving backwards through the layers to train the model. Together, forward propagation and backpropagation enable a neural network to make predictions and correct for any errors . Over time, the algorithm becomes gradually more accurate.
Deep learning requires a tremendous amount of computing power. High-performance graphical processing units (GPUs) are ideal because they can handle a large volume of calculations in multiple cores with copious memory available. Distributed cloud computing might also assist. This level of computing power is necessary to train deep algorithms through deep learning. However, managing multiple GPUs on premises can create a large demand on internal resources and be incredibly costly to scale. For software requirements, most deep learning apps are coded with one of these three learning frameworks: JAX, PyTorch or TensorFlow.
Deep learning algorithms are incredibly complex, and there are different types of neural networks to address specific problems or datasets. Here are six. Each has its own advantages and they are presented here roughly in the order of their development, with each successive model adjusting to overcome a weakness in a previous model.
Convolutional neural networks (CNNs or ConvNets) are used primarily in computer vision and image classification applications. They can detect features and patterns within images and videos, enabling tasks such as object detection, image recognition, pattern recognition and face recognition. These networks harness principles from linear algebra, particularly matrix multiplication, to identify patterns within an image.
CNNs are a specific type of neural network, which is composed of node layers, containing an input layer, one or more hidden layers and an output layer. Each node connects to another and has an associated weight and threshold. If the output of any individual node is above the specified threshold value, that node is activated, sending data to the next layer of the network. Otherwise, no data is passed along to the next layer of the network.
CNNs are distinguished from other neural networks by their superior performance with image, speech or audio signal inputs. Before CNNs, manual and time-consuming feature extraction methods were used to identify objects in images. However, CNNs now provide a more scalable approach to image classification and object recognition tasks, and process high-dimensional data. And CNNs can exchange data between layers, to deliver more efficient data processing. While information might be lost in the pooling layer, this might be outweighed by the benefits of CNNs, which can help to reduce complexity, improve efficiency and limit risk of overfitting.
Recurrent neural networks (RNNs) are typically used in natural language and speech recognition applications as they use sequential or time-series data. RNNs can be identified by their feedback loops. These learning algorithms are primarily used when using time-series data to make predictions about future outcomes. Use cases include stock market predictions or sales forecasting, or ordinal or temporal problems, such as language translation, natural language processing (NLP), speech recognition and image captioning. These functions are often incorporated into popular applications such as Siri, voice search and Google Translate.
RNNs share parameters across each layer of the network and share the same weight parameter within each layer of the network, with the weights adjusted through the processes of backpropagation and gradient descent to facilitate reinforcement learning.
RNNs use a backpropagation through time (BPTT) algorithm to determine the gradients, which is slightly different from traditional backpropagation as it is specific to sequence data. The principles of BPTT are the same as traditional backpropagation, where the model trains itself by calculating errors from its output layer to its input layer. BPTT differs from the traditional approach in that BPTT sums errors at each time step, whereas feedforward networks do not need to sum errors as they do not share parameters across each layer.
An advantage over other neural network types is that RNNs use both binary data processing and memory. RNNs can plan out multiple inputs and productions so that rather than delivering only one result for a single input, RMMs can produce one-to-many, many-to-one or many-to-many outputs.
There are also options within RNNs. For example, the long short-term memory (LSTM) network is superior to simple RNNs by learning and acting on longer-term dependencies.
However, RNNs tend to run into two basic problems, known as exploding gradients and vanishing gradients. These issues are defined by the size of the gradient, which is the slope of the loss function along the error curve.
Autoencoders work by encoding unlabeled data into a compressed representation, and then decoding the data back into its original form. Plain autoencoders were used for a variety of purposes, including reconstructing corrupted or blurry images. Variational autoencoders added the critical ability not just to reconstruct data, but also to output variations on the original data.
There are disadvantages to autoencoders. The training of deep or intricate structures can be a drain on computational resources. And during unsupervised training, the model might overlook the needed properties and instead simply replicate the input data. Autoencoders might also overlook complex data linkages in structured data so that it does not correctly identify complex relationships.
GANs train themselves. The generator creates fakes while the discriminator learns to spot the differences between the generator's fakes and the true examples. When the discriminator is able to flag the fake, then the generator is penalized. The feedback loop continues until the generator succeeds in producing output that the discriminator cannot distinguish.
c80f0f1006