In machine learning, a neural network (NN) or neural net, is a computational model inspired by the structure and functions of biological neural networks. A neural network consists of connected units or nodes called artificial neurons, which loosely model the neurons in the brain. These are connected by edges, which model the synapses in the brain. Each artificial neuron receives signals from connected neurons, then processes them and sends a signal to other connected neurons. The "signal" is a real number, and the output of each neuron is computed by some non-linear function of the totality of its inputs, called the activation function. The strength of the signal at each connection is determined by a weight, which adjusts as part of the training process. Groups of neurons are aggregated into layers. Each layer performs a transformation on its inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), typically passing through multiple intermediate layers (hidden layers). A network is typically called a deep neural network if it has at least two hidden layers. Deep neural networks are capable of learning sophisticated hierarchical representations. Training neural networks is a compute-intensive process, accelerated by the use of graphics processing units (GPUs), and large datasets.
In reality, such textures and outlines would not be represented by single nodes, but rather by associated weight patterns of multiple nodes. Architectural innovations such as convolutional neural networks (CNNs) significantly improved performance in computer vision tasks, while recurrent neural networks (RNNs) enabled modeling of sequential data such as speech and time-series information. Transformer architectures introduced attention mechanisms that allow neural networks to model long-range dependencies in data and have been the basis of large language models. Artificial neural networks are used for a myriad of tasks including chatbots, large-scale text, image, and video generation, and robotics.
History
Mathematical foundations Deep neural networks are based on statistics developed over 200 years ago. The simplest kind of feedforward neural network (FNN) is a linear network, which consists of a single layer of output nodes with linear activation functions; the inputs are fed directly to the outputs via weights. The sum of the products of the weights and the inputs is calculated at each node. The mean squared errors between these calculated outputs and the given target values are minimized by adjusting to the weights. This technique is the method of least squares or linear regression. It was used to find a rough linear fit to a set of points by Legendre (1805) and Gauss (1795) for the prediction of planetary movement.
Perceptrons Computers are based on John von Neumann's model. They execute explicit lists of instructions with access to memory to record their changing state. Neural networks instead originated from efforts to model information processing in biological systems via connectionism. Unlike the von Neumann model, connectionist computing does not separate memory and processing. Warren McCulloch and Walter Pitts (1943) considered a non-learning computational model for neural networks. This model paved the way for research to split into one branch focused on biological processes and another focused on artificial intelligence. McCulloch and Pitts also developed mathematical models of artificial neurons capable of representing logical functions. In the late 1940s, D. O. Hebb proposed a learning hypothesis based on neural plasticity that became known as Hebbian learning. It was used in many early neural network experiments, such as Rosenblatt's perceptron and the Hopfield network. Farley and Clark (1954) used computational machines to simulate a Hebbian network. Other neural networks computational machines were created by Rochester, Holland, Habit and Duda (1956). In 1958, psychologist Frank Rosenblatt described the perceptron, one of the first implemented neural networks, funded by the United States Office of Naval Research. R. D. Joseph (1960) mentioned an earlier perceptron-like device by B. G. Farley and W. A. Clark of the MIT Lincoln Laboratory; however, according to Joseph, "they dropped the subject." The first perceptrons did not have adaptive hidden units. However, Joseph (1960) discussed multilayer that did. Rosenblatt (1962) cited and adopted these ideas, crediting work by H. D. Block and B. W. Knight. However, these early efforts did not lead to a working learning algorithm for hidden units, i.e., deep learning. The perceptron raised public excitement in neural networks, causing the US government to drastically increase funding. This contributed to "the Golden Age of AI", fueled by the optimistic claims made by computer scientists regarding the ability of perceptrons to emulate human intelligence.
Historical foundations and the Dartmouth proposal Artificial neural networks were identified as a promising direction for artificial intelligence research in the 1955 proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Neural network models initially faced major limitations. Hardware constraints limited network size and training efficiency, while theoretical understanding of learning algorithms remained incomplete. Many models used single-layer perceptrons, which were restricted to solving linearly separable problems. These limitations were highlighted in the book Perceptrons by Marvin Minsky and Seymour Papert, which deflated interest during the late 1960s and 1970s.
… excerpt ends here. Continue reading the full article.
